Thilak

1. Lebara Datalake – Senior Data Engineer and Architect

Client                   : LEBARA-LTD

Duration             : May 2022 – Current

Role                     : Team Lead & Azure Data Engineer

Technologies     : Azure Databricks, Azure Data factory, ADLS, Spark, GIT, CICD, Pycharm, Synapse

Description:

LEBARA, a telecom company, where we are processing the data of network subscribers who use the network with different services like voice, data etc.

Roles & Responsibilities:

• Designing the ADF pipelines for dataflow from ingestions to data processing using Azure Databricks and many other activities in ADF and loads data to Synapse using Stored procedures.
• With Pycharm, developed ADB notebooks using PySpark and ADF pipelines for designing.

·        Used GIT as code Repository and CICD for deployment of ADB code and ADF pipelines to cloud.

·        Implemented may code Optimizations to save Time and Cost

·        Implemented Cluster Tagging to all Databricks notebooks and Data Factory linked services to easily identify the resources consumption in cost and time in Cost Analysis portal

·        Implemented Data Life Cycle Management Rules in Azure DataLake Gen2 Blog storage to delete the data files which are older than given period of time which saves the storage cost and also processing cost & time to process huge unnecessary data.

·        Part of Architect to design Data pipelines and to save the cost and time.

1.  Azure Cloud Migration – Lead Data Engineer

Project                :  Mysecure Workstation

Duration             : January 2020 – April 2022

Role                     : Team Lead & Azure Data Engineer

Technologies     : Azure Databricks, Azure Data factory, ADLS, Spark, GIT, CICD

Description:

Migration of Project from On-premise to Azure cloud, code from Spark Scala & Hive scripts to Azure Databricks, Talend jobs to Azure Data Factory for batch processing, new Databricks jobs for Streaming data to process.

Developed ADB jobs and ADF pipelines. Used App Insights for logging.

Used GIT as code Repository and CICD for deployment of ADB code and ADF pipelines to cloud.

Roles & Responsibilities:

·        Involved in all migration plan decision making meetings for the migration to Azure Cloud.

·        SME and Data Engineer for the cloud migration.

·        Developed Azure Databricks Notebooks for code migration from on-premise Hadoop ecosystem.

·        Developed Azure Data Factory pipelines for Orchestration of notebooks.

·        Lead the team and monitoring all team members tasks, motivating and guiding them.

·        Responsible to prepare & deliver Weekly & Sprint reports.