THILAK, AZURE DATA ENGINEER | Bench Resource on Contract

AZURE DATA ENGINEER

Location : BANGALORE, India

Experience : 9 Year

Rate: $21 / Hourly

Availability : Immediate

Work From : Offsite

Category : Information Technology & Services

Shortlisted : 1

Total Views : 60

Key Skills

AZURE DATA ARCHITECT KAFKA PYSPARK SQL PYTHON JAVA AWS ECLIPSE CI/CD TALEND

Discription

THILAK

Senior Data Engineer

Bangalore

Summary:

Extensive 9 years of professional experience in the areas of Data Engineering, Data analysis, Software Life Cycle Management, Microsoft Azure Cloud, Functional Testing, Agile/DevOps, Big data ecosystem with sound knowledge on banking domain.
Having experience on Hadoop framework components like Hive, MapReduce, Spark, Kafka,, Scala.
Around 5 years of experience in Bigdata Hadoop and Spark technologies stack.
Hands-on experience and sound knowledge in Microsoft Azure Cloud Resources like Azure Databricks, Azure Data Factory, ADLS, Synapse Database etc
Having good experience in writing PySpark, scala, SQL, Hive queries and Hive tables management.
Hands on experience on Talend tool to run the workflow jobs
Coordinated with clients in understanding the requirements and providing on time delivery as per the client requirement in their specified sprint
Able to adapt quickly to new requests in all phases of SDLC, co-ordinate with the Infrastructure team in case of maintenance and unavailability of application.
An effective communicator with excellent analytical capabilities, with a self-motivated and positive approach within an organization.

Skills:

Big Data / Hadoop : Spark, Hive, Mapreduce, Kafka, Yarn

Scripting & Programming Languages : Pyspark , SQL, Scala, SparkQL, Python, Java

Cloud Ecosystems : Microsoft Azure [ADB, ADLS, ADF etc], AWS Fundamentals

Tools : Talend, Eclipse, CI/CD, GIT, Visual Studio, Pycharm, Jira

SDLC : Agile (Scrum), Waterfall.

Work Experience:

Company : COnfidential Engineering

Client : LEBARA-LTD

Duration : May 2022 – Current

Role : Team Lead & Azure Data Engineer

Technologies : Azure Databricks, Azure Data factory, ADLS, Spark, GIT, CICD, Pycharm, Synapse

Description:

LEBARA, a telecom company, where we are processing the data of network subscribers who use the network with different services like voice, data etc.

Roles & Responsibilities:

Designing the ADF pipelines for dataflow from ingestions to data processing using Azure Databricks and many other activities in ADF and loads data to Synapse using Stored procedures.
With Pycharm, developed ADB notebooks using PySpark and ADF pipelines for designing.
Used GIT as code Repository and CICD for deployment of ADB code and ADF pipelines to cloud.
Implemented may code Optimizations to save Time and Cost
Implemented Cluster Tagging to all Databricks notebooks and Data Factory linked services to easily identify the resources consumption in cost and time in Cost Analysis portal
Implemented Data Life Cycle Management Rules in Azure DataLake Gen2 Blog storage to delete the data files which are older than given period of time which saves the storage cost and also processing cost & time to process huge unnecessary data.
Par