Purushothaman, Senior Data Engineer | Bench Resource on Contract

Purushothaman (RID : 210ulpdorzjw)

Senior Data Engineer

Location : Ahmedabad

Experience : 15 Year

Rate: $20 / Hourly

Availability : 1 Week

Work From : Offsite

Category : Information Technology & Services

Shortlisted : 2

Total Views : 49

Key Skills

PySpark python java AWS Snowflake SQL SageMaker Airflow ETL Big data

Discription

Total 16+ years of experience in various domain knowledge like Banking, telecom, travels, and healthcare and primary responsibilities are to understand business requirements, system analysis, propose new or change architecture, involving in project development and implementation etc.
Primary responsibilities included designing and implementing data processing workflows, building new data pipeline, managing data quality, monitoring system performance, and troubleshooting issues.
Strong working experience in high level programing language in Scala, Python and Java
Strong working experience in PySpark with AWS EMR Environment
Very good working experience in AWS like AWS EC2, ECS, RDS, Dynamo DB, EMR, S3, API Gateway, Lambda, Cloud watch, Glue
Very good working experience in Snowflake and Airflow scheduler
Good Knowledge on UNIX shell scripting.
Experience in understanding business requirements for analysis, design & development of applications.
Experience leading cross functional teams across a large organization in systems analysis, defining technical requirements, and performing high level design for highly complex solutions mainly on Bigdata and Hadoop Analytics

Project Details

Title : AstraZeneca

Duration : 18 (Month)

role and responsibileties :

Design, development and implemented of performant ETL pipelines on AWS EMR Environment and Integration of data storage solutions in PySpark especially with AWS S3 storage.
Design and Developed ETL Processes in AWS Glue to migrate Campaign data from external sources like S3 files, parquet and csv Files into AWS Athena.
Perform project management activities including project documentation, business requirements and project tracking
Performed end-to-end architecture and implementation assessments of numerous AWS services (including AWS EMR, AWS Glue, AWS EC2, AWS Lambda and Amazon Simple Storage Service)
Extract the data from various formats like CSV, JSON and convert to parquet format and loaded to AWS S3 bucket using AWS glue and PySpark

Description :

AstraZeneca specialty healthcare services with a science-led, patient-focused pharmaceutical company and focused on creating genuinely innovative medicines and improving access and to deliver the greatest benefit to patients, healthcare systems and societies globally. AZ Brain is project intended to generate insights from Real world evidence data and surface the insights that will be used for generating the next best action. Sonata software, India providing software solution and support to AstraZeneca. We are migrating SQL data to AWS cloud services through AWS EMR environment and provides support and software solution to AstraZeneca. Also, we shared processed data to build ML model application for disease prediction through AWS SageMaker environment

Title : Enterprise data management system (EDMS)

Duration : 18 (Month)

role and responsibileties :

Financial Services is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market. Sonata software, India providing software solution and support to PennyMac. We are migrating on-premise data to cloud with AWS cloud services and snowflake and software support and solution. Also, we providing migrated data to end user.

Description :

Design, developed and implemented to extract data from Microsoft sql server to S3 bucket using Lambda function, step function, DynamoDB, CloudWatch and deployed to AWS production environment.
Design, Developed and implemented the ETL Processes using AWS Glue to migrate Campaign data from S3 datelake to Snowflake and AWS Athena
Deploy fully operational data warehouse solutions into production on Snowflake
Created Snowflake data modeling and ELT using Snowflake SQL, implementing complex stored procedures and best practices with data warehouse and ETL concepts
Utilized Python Libraries like Boto3, Pandas, NumPy, pyarrow for developing AWS lambda application to extract data from various source files and loaded into AWS S3 bucket
Preparation of the use cases best suited for understanding the job execution and performance tuning parameters
Created python script that to optimizes performance of AWS services used by application teams and provide code-level application security for clients (IAM roles, credentials, etc.)
Extract the data from various formats like CSV, JSON and convert to parquet format and loaded to AWS S3 bucket using AWS glue and AWS Lambda
Developed Spark based application using high level programming language python and PySpark

Title : Cyrus Next Generation (CNG)

Duration : 25 (Month)

role and responsibileties :

Design, developed and implemented to ingest different types of data from On Premise Database to data lake with python and Airflow
Extracted the data with various formats like CSV, JSON loaded to AWS S3 bucket
Data extraction from AWS S3 and loaded to Snowflake with Python and airflow
Developed PySpark based application using python

Perform code peer reviews, prepare and review test data

Description :

Cyrus Next Generation enables traders by allowing package level control and automating millions of price moves based on capacity, demand and margin. Re-platform and develop CYRUS to deliver scalability and greater capabilities supporting both business growth and evolving business model. The System able to scale in order to handle predicted and unpredicted changing demand over time of booking throughput and package availability. The System shall provide the capability to easily increase capacity within the boundaries of agreed licensing with cloud provider(s) and Source Markets