Data Engineer
Location : Chennai, India
Experience : 5 Year
Rate: $15 / Hourly
Availability : Immediate
Work From : Offsite
Category : Information Technology & Services
Jegath S
Senior Data Engineer
Profile
An experienced data engineer who is results-oriented, resourceful, and able to solve problems.
Work Experience
Skills/ Tools : Cloud(AWS, Google Cloud, Azure [Certified Data
Engineer] ), Python, Pyspark, Bigquery, Apache Airflow, Java, Javascript, MySQL, Git, Terraform, Shell scripting, YAML, Android, Nodejs, Flask framework, Docker, MongoDB, React, Datadog, Dataproc, Google Cloud Data Fusion, AWS Athena, Azure Data
Factory, Talend Data Studio.
Senior Data Engineer
Wise Analytics | May 2021 - March 2023
Used pyspark and Google Dataproc to create a data validation pipeline that validates CSV data and pushes valid records to
bigquery tables Creating and managing various pipelines using google cloud composer (managed airflow) Optimizing queries to reduce the time to load as well as the overall cost
Created jupyterhub environments for multiple users using docker and used terraform to make the deployment faster, easier, and error- proof.
Leading a growing team currently consisting of 6 people to create, maintain, manage airflow pipelines, monitor google cloud
infrastructure using data dog, create a custom portal where clients can see their data Created an automation tool that reduced the time to create a certain
kind of DAG in the airflow by 70% Migrated composer environment to the newest version
Created GitLab CI/CD pipeline to sync the changes from GitLab repo
to composer bucket and managed all the dags
Data Engineer
Adfolks LLP | Feb 2020 - May 2021
Developed a flask application to enrich or collect user information
from different public API’s
Created another python application to consume data which is sent
to Azure event hub
Another python based application which will run periodically in
Kubernetes to generate an invoice
A React-based application to view all the enriched or collected
details which include ACL level access for users
Containerized all these applications using docker and deployed in
Azure Kubernetes
Projects
Customer Enrichment
Extract data from multiple sources like Dynamic 365, Flat files(CSV, Excel, XML), SQL DB to Data Lake
Transform data according to client specifications
Load transformed data to the data warehouse(synapse)
Used Azure Data Factory as job orchestrator
Raw data from the Autoline Kerridge system is extracted and loaded to AWS S3 bucket
Using AWS Athena transformation are done on S3 files leveraging AWS glue for metadata
Transformed data is stored on AWS cloud data warehouse redshift
Job orchestration is done on Talend ETL tool with custom java scripting
Reduced the workload and maintenance of the project drastically by converting 50+ job architecture to 1 job
architecture
Azure Data warehousing
AWS Data Migration
Developer
Tata Consultancy Services | July 2018 - Feb 2020
Developed a Python application that sends browser data (history) to the server automatically at periodic
intervals
Done Exploratory data analysis (EDA) on the collected data to extract some useful information for the
organization
Developed a Django REST application that suggests new learning content and shows various insights to fresher
which are predicted by a deep learning model
Integrated OpenCV in android application to detect a jacquard
Automatic verification and validation of the incorporated values from the detected card
Android Application to help blind people using Face Recognition, emotion detection, and object detection, with a
single snap
Analyze the objects, detect, recognize and find out the emotions of the people standing in front of the camera
A portable python package, to run python without installing it to the system
It can work with just running a single file for installing any number of packages that we specify
Set up a portable python environment, that can work even if we copy it elsewhere
Projects
Browser Analytics
Automatic Jacquard Validation
Other/ Personal projects
Gods Eye
AIOP
Achievement and Awards
Certified Azure Data Engineer
Gained learning achievement award under organization level
Best outgoing student of the year 2018 from college
Academic History
Bachelor of Computer Application | 2015- 2018
St Michales HSS Kaduthuruthy (XII Grade) | 2013- 2015
St Anne's HSS Kurianad (X Grade) | 2013
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
