DATA ENGINEER AWS
Location : PUNE, India
Experience : 10 Year
Rate: $17 / Hourly
Availability : Immediate
Work From : Offsite
Category : Information Technology & Services
Rishi
Primary Skills:
· Data Architecture and Modelling.
· Bigdata development using Pyspark, Hive, Airflow, YARN, HDFS.
· AWS: Glue, EMR, S3, Lambda
· Programming using Python and Java, Plsql.
· Data Streaming using Kafka, Nifi, Spark Streaming.
· ETL Pipeline Designing using Informatica, Talend.
· Automation Using Unix Shell Scripting and Python
· Banking
· Compliance
· Telecom
· Date of Birth: March 1992
· Address: Pune
· More than 10 years of experience in Data Architecture and Engineering, specializing in building smart solutions for businesses.
· Expert in designing and developing solutions for Big Data, ETL, Streaming, and Analytics, using both Agile and waterfall methods.
· Extensive knowledge in setting up data warehouses, marts, and lakes from start to finish.
· Leader of teams handling critical projects across different business areas, with a track record of successful deliveries.
· Proven expertise in Technology Consulting for data-driven projects, including AWS experience in Lambda and Glue, EMR for cloud-based solutions.
· Contributed to optimizing processes and solutions, resulting in operational and financial benefits for businesses.
· Implemented solutions for metadata management, audit control, and data security to meet regulatory standards.
Organization
Designation
From Date
To Date
Confidential
Independent Consultant
Dec 2020
Till data
Confidential
Tech Lead
Feb 2019
Dec 2020
Confidential
Senior Software Engineer
Nov 2016
Feb 2019
Confidential
Software Engineer
Sept 2013
Apr 2016
Educational Background:
· B.E in Information Technology from University of Pune in 2013 with 61.06%.
· Diploma in Computer Engineering from MSBTE in 2010 with 75.08%.
· X from State board of Maharashtra in 2007 with 83.23%.
Project Name
OneApp – Silver layer Architecture
Technology
Data Valut
Client
Deutsche Telekom
Project aims at architecting a layer from raw data(bronze layer), for data readiness of different mlops projects and Gold layer re-implmentation.
· Designed and architected a real-time data transformation solution using Spark Streaming on top of the OneApp DE Kafka pipeline.
· Developed a data storage strategy that organizes Events Data into specific event types, NATCO, and date partitions in the S3 data lake, ensuring efficient data retrieval and analysis.
· Implemented a robust schema management system using Schema Registry to maintain data consistency and integrity.
· Designed and executed a data backfilling job to load historical data for the past two years from the OneApp S3 Data Lake, ensuring a comprehensive dataset for analysis.
· Integrated UA (User- Attributes), SA (Session Attributes), and Device data into the Silver Layer architecture, aligning with data ownership and consumption requirements.
· Designed and implemented a Spark Streaming job to update UA, SA, and Device Data, following the Data Vault model for data warehousing and management.
Project Name
OneApp - B2B prediction use case and dashboards
Technology
Hive,Kafka,Nifi,Aws services, Spark
Client
Deutsc
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
