Lokesh Reddy
PROFESSIONAL SUMMARY
Experienced Data Engineer adept at architecting and deploying scalable ETL pipelines using AWS Glue, Lambda, and DMS, complemented by deep expertise in PySpark, Python, and SQL. Proven track record in optimizing cloud-based data workflows, ensuring seamless integration, efficient data migration, and delivering actionable insights to drive business outcomes.
SKILLS
- Frameworks and Tools: Jupyter Notebook, VSCode, SCT, SSMS, PGAdmin
- Programming Language: Python, R, PySpark
- Python Modules: Pandas, NumPy, NLTK, sklearn, tika, matplotlib, seaborn
- AWS Services: Data Migration Services, S3, Lambda, Athena, EC2, CloudWatch, Step Functions, Code Commit, Sage Maker
- SQL: MySQL, DB2, Postgres
- CI/CD: AWS Code commit, Jenkins
- Data Warehouse: Snowflake
EXPERIENCE
Capgemini, Hyderabad – Senior Consultant
Feb 2022 – May 2023
PROJECT 1 - AWS DATA ENGINEER
Responsibilities:
- Spearheaded the migration of SQL stored procedures from SQL Server Management Studio (SSMS) to PostgreSQL, ensuring compatibility with AWS Aurora.
- Expertly leveraged AWS Glue with PySpark to conduct robust ETL processes for the migrated stored procedures.
- Developed sophisticated conversion code using Spark SQL and PySpark DataFrame API, ensuring seamless data transformation and loading.
- Utilized AWS Schema Conversion Tool (SCT) for streamlined table creation in the target AWS Aurora environment.
- Orchestrated the sequenced execution of Glue jobs using AWS Step Functions, mirroring step-wise execution protocols from SSMS.
- Developed a modular AWS Glue job framework to efficiently handle similar procedures, minimizing redundancy by leveraging parameterized execution.
PROJECT 2 - Data Engineer Responsibilities:
- Developed diverse data layers utilizing big data and AWS technologies.
- Managed AWS REDSHIFT data transfer to Data Warehouse through CI/CD pipelines.
- Established YAML configurations for each entity to transition data from Redshift to the AWS S3 serving bucket. These configurations are archived in a Github repository.
- Leveraged Jenkins for YAML configuration deployment, enabling Airflow DAGs to facilitate data transfer.
- Executed both FULL LOAD and DELTA LOAD data transfer strategies.
- Transferred data from the S3 serving bucket to the Snowflake Data Warehouse using Snowflake's PIPE, Stage, Stream, and Procedures.
- Automated data ingestion from S3 to the Snowflake stage using Snowpipe.
- Upon transferring data to the Snowflake stage, a stream is initiated atop the stage data, subsequently moving it to the main table. This process is orchestrated by a Snowflake procedure activated at 30-minute intervals.
- Crafted specific Snowflake procedures tailored for Full Load and Incremental Load operations.
Tata Consultancy Services, Hyderabad – AWS Data Engineer
Sep 2016 – Feb 2022
AWS Data Engineer Responsibilities:
- Developed diverse data layers leveraging big data and AWS solutions.
- Crafted ETL pipelines for Data Lake and Data Mart utilizing AWS Glue and Lambda.
- Achieved a 99.9% uptime for data pipelines, processing data from three key sources with PySpark, Lambda, S3, and Python.
- Proficient in AWS services including AWS Glue, DMS, Lambda, Step Functions, RDS, Athena, Cloud Watch, EC2, and API Gateway.
- Orchestrated the migration of data from on-premise systems (MYSQL, IBM DB2, Filesystem) to AWS S3 using DMS. • Constructed and maintained real-time data streams.
- Engineered data extraction, transformation, and loading processes, either through Step functions
or bespoke data pipelines. • Deployed SCD1 and SCD2 methodologies via DMS.
- Enabled continuous data capture (CDC) from original data repositories using DMS.
- Demonstrated expertise in crafting and optimizing SQL queries. • Drafted deployment strategies for
various scenarios and played a pivotal role in release management.
- Engaged with distributed data infrastructures for storage and computation.
- Managed data transfer from source systems to relational databases for visualization.
Data Analyst Responsibilities:
- Identify valuable data sources and automate collection processes.
- Undertake pre-processing of structured and unstructured data.
- Perform Exploratory Data Analysis on data for better understanding.
- Extensive use of Pandas library on data.
- Analyse large amounts of information to discover trends and patterns.
- Build predictive models and machine-learning algorithms.
- Perform Data Modelling on structured and unstructured data.
- Apply CART models on data like Decision Tree, Random Forest, AdaBoost, SVM, KNN.
- Worked on text analytics like topic modelling, sentiment analysis.
- Optimization of data on machine learning models.
EDUCATION
POST GRADUATION DIPLOMA (Data Science) from Manipal Academy of Higher Education, Karnataka
JUL 2018 - JUN 2019
B. Tech (Computer Science) from G. Pulla Reddy Engineering College (Autonomous). OCT 2012 - APRIL 2016
HONORS and ACHIEVEMENTS
- State Level Cricket Player
AWS Redshift
Location : Hyderabad, India
Experience : 6 Year
Rate: $14 / Hourly
Availability : Immediate
Work From : Offsite
Category : Information Technology & Services
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
