AWS Data Engineer
Location : Gurgaon
Experience : 9 Year
Rate: $16 / Hourly
Availability : Immediate
Work From : Offsite
Category : Information Technology & Services
SUMMARY
9+ years experienced Data Solution Architect with a prolific record managing ETL, Data Mining,Data Modeling and Data Warehousing operations within an enterprise. Proficient in leading a team of ETL Developers & Data Analysts to improve data management. Adept at developing end-to-end solutions to migrate & integrate data in SAP & Cloud platforms.
PROFESSIONAL EXPERIENCE
Data Solution Architect-
Ecosystem Consulting
An end-to-end solutions provider in delivering Data Solutions
Technology Stack: Python,sql,Tableau,Snowflake,Mysql,ETL,CI\CD,Handoop,Scala,Kubernetes,Airflow,Azure data Factory,Postgresql,Scala,Redshift,Athena,Tempo,Data Modeling,Neo4J,Databricks,Logic App,Grafana,Loki,Pyspark,Scala,Mongo DB,Power BI,Quicksight,SAP HANA,Data lake,Azure Synapse,Spark,DBT cloud,Streamsets
Project-1 : Data Solutioning,ETL ,Data Warehousing,Data Architecture design,Dashboarding
· Working as Big Data and Cloud Architect with business partners, application product owners to understand detailed business requirement, and translating them to technical requirement.
· Designed data warehouse and Data Lake solution along with data processing Pipeline using PySpark using GLUE, Lambda, EMR,Step Function,Glue Workflow,Glue Data catalogue ,Athena and Redshift. Entire cloud infrastructure created using AWS Cloud formation.
· Performed Data Modeling on Aurora MYSQL, Redshift, DynamoDB for transactional and analytics need .
· Designed and developed a pipeline using Kinesis Data Stream ,Kinesis Delivery Stream to load data to data lake from Aftermarket portal.
· Involved on hands on development and configuration of data processing using Pyspark on AWS Glue using Pycharm,Glue studio and Jupiter notebook
· Designed and Implemented the Data Migration from On Premise to Cloud using DMS
· Created various dashboards using cloud watch for monitoring various stats of the system .
· Implemented all PACCAR standard security policy in AWS Cloud including SSL security at application label.
· Designed Data model for the warehouse using Erwin followed by physical database creation with required standards
· Developed event based ETL load using S3, Lambda, AWS Glue, and Redshift for secondary manufacturing module.
· Designed and developed data catalogue using Glue Crawlers and used them in on Athena for analytics and Glue Jobs for ETL use.
· Implemented CICD for AWS Glue using AWS CF, AWS CODE Build,AWS Code Deploy.
· Worked and delivered critical on-premises to cloud modernization projects as part of AWS cloud migration journey.
· Define and standardize and developed - HA and DR strategy for the platform.
· Automated infrastructure shutdown and startup of on demand systems using SSM which business could start and stop from there user screen.
· Automated AWS Key and password rotation using AWS Lambda,AWS Secretmanger.Cloudwatch event.
· Designed and developed data lake on AWS S3 and integrated data from various systems.
· Designed and developed data ingestion framework using Talend Big data to ingest data to data lake from various data sources.
· Designed and Implemented the history Data Migration from On Premise database to Cloud using DMS.
· Created various dashboards using cloud watch for monitoring various stats of the system (that includes EC2/RDS)
· Implemented all PACCAR standard security policy in AWS Cloud including SSL security at application label.
· Designed Data model for the warehouse using Erwin followed by physical database creation with required standards and indexes for better performance.
· Event based ETL load designed using Pyspark and Step function EMR to process data on spark cluster and loads to Aurora database.
· Designed and Implemented CICD on AWS for Talend code using Jenkin for automated and faster code promotion which also integrated with
automated test case running on the CICD pipeline. Azure VSTS and Azure Code pipeline are used to integrate with Jenkins and Talend CI Builder/maven/TalendMetaservlet along with Jupiter (Cognizant's Internal tool for testing automation)
· Configured AWS EMR to use by Talend for Data Ingestion.
· Configured all AWS services like (EC2/RDS/REDSHIFT/IAM/SSM/CLOUDWATCH/ECS etc) using cloud formation.
· Automated entire infra backup based on tags using AWS Backup.
· AWS Cloud watch ,SNS: for system and process monitoring and alerts
Remote,India
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
