ANURAAG GUPTA

Senior Consultant

Synopsis

• 4+ years of hands on experience in development of Bigdata Technologies and AWS CloudTechnologies.
• Currently working as a Data Engineer at Healthcare Domain in with hive language and Spark (Pyspark) in AWS cloud platform for data storage and analysis. Also, working on CI/CD pipeline (Jenkins) with GitHub, ServiceNow integration.
• Comprehensive understanding of Amazon S3, VPC, IAM, EMR, CloudWatch, Lambda and other attributes of Amazon Web Services.
• Demonstrated excellence while working as a Senior Bigdata Engineer/Data Engineer at Mercedes Benz R&D Centre in Automotive domain through IBM in a project titled as Proactive Sensing and Data Vault with Hive, Spark-Streaming, Oozie as the main technologies.
• Eminently engaged in the Projects using Hadoop Ecosystem as well as Hadoop Architecture and its various components such as MR, HDFS, Hive, Pig, Oozie.
• Proficient in DWH, & programming languages Python, also in programming techniques and industry standard software documentation techniques.

Core Competencies

• Debugging
• Analytical, logical and problem-solving skills

• Teamwork
• Testing and Problem Solving

• Adaptability
• Migration of OnPrem applications to AWS

Tools and Technologies

• Hadoop
• Oozie
• Hive
• EC2
• AWS S3

• Pig
• CloudWatch
• EMR
• AWS Lake formation
• Crawler

• AWS IAM roles
• AWS Athena
• Airflow
• Lambda
• RDS

Responsibilities

• Design and develop Migration framework for Hadoop to AWS
• Resolved Application Performance issues and EMR cluster optimization
• Efficient in writing the Pig Scripts to reduce the job execution time
• Responsible for leading a team of 5 people at offshore and co-ordinate with onshore and clients
• Review the developed code and make sure it adheres to the design, standards and guidelines of the client and Virtusa
• Conduct meetings with business and discuss the daily status
• Conduct daily stand up agile sprint calls to update the status
• Converting SQL scripts to Hive Scripts
• End to end automation on pulling data from various sources using AWS Glue.
• Migrating Hive, Pig, Shell and Spark scripts from Hadoop to EMR

Organizational Responsibilities

• Involved in handling interviews internally and lateral for other projects
• Contribute to the development of the Software Engineering competency

Educational Qualifications

• Graduation in B.Tech from JNTU

Achievements & Certifications

• AWS Certified Solutions Architect – Associate

Present Employer

Role

AWS Senior Consultant

Technology

AWS, Hive, Pig, Python, SQL

Description

Migration from On-Prem to AWS Cloud

Responsibilities

• Cloud Migration for existing application migration from Hadoop On-Premise to AWS
• Cloud system using AWS services like VPC, EMR, S3, MWAA, Glue and Big Data analytic tools
• Created Hive external tables on top of S3 data for different data file formats, adding partitions, coding as per the requirements using Hive (HQL), Java (UDFs) and Shell scripting.
• Root cause analysis of both script and data issues
• Analyzed and developing Pig queries and Spark Queries for migration from Hadoop to AWS.
• Analyzing the flow of execution and Develop automation scripts for migration of Hive, Pig, Spark, Data science jobs and parameterization of the scripts in different

environments

• Executed Apache Airflow DAG for the jobs to execute in a scheduled time.
• Analyzed pattern requirements, create user stories and sub-tasks and distribute story points and expected hours for the subtasks on JIRA.
• Re-Write several slow performing HQL queries or application to fast and cost efficiently performing during Post-Migration.
• Analyzed the Jobs In detail to determine right instance/cluster type (Storage/IO/Compute/Memory Optimized) for cost efficient performance on AWS EMR cluster
• Provided mentorship to fellow Developers
• Actively participated in the sprint planning meetings in the Agile scrum and explained the issues in laymen terms to help BSAs understand
• Designed and deployed EC2 auto-scaling.
• Working on S3 storage life cycle management to move the objects to glacier storage when it meets life cycle policy.
• Managing IAM role, IAM user groups and access keys.
• Deploy infrastructure to multiple environments using cloud formation.

Experience Prior to Virtusa

Client: Benz

Role

System engineer

Technology

AWS

Description

Migration from On-Prem to EMR

Responsibilities

• Developing cloud solution for inbound live stream. Data flow: API gateway, Lambda, SNS, SQS, S3 and Glue database.
• Deploying cloud solution using Cloud Formation Template, infrastructure as code.
• Creation Jenkins pipeline for AWS cloud resources deployment.
• Developed Lambda (Python) for POST messages from API gateway.
• EMR, Glue database and cloud watch schedule setup for Batch and Delta loads.
• Ensuring secure data security by using encryption – KMS
• Built data pipeline over private network using – Private API gateway and VPC end points.
• Configuring Lambda trigger, Security group, environment variables.
• Defining IAM roles and IAM policies. Restricting user access to specific resources.
• Securing environment by keeping components within VPC and communicating service over private network using VPC endpoints.
• Jenkin pipeline creation and deploying infra and code through Urban code deploy.