Data Engineer
Location : Delhi, India
Experience : 8 Year
Rate: $16 / Hourly
Availability : Immediate
Work From : Any
Category : Information Technology & Services
PROFESSIONAL SUMMARY
TECHNICAL SKILLS
Languages
Scala
Tools
Spark, Databricks, Hadoop, Map Reduce, Jenkins, Jira, Hive, Maven, Dremio, Nifi, Azure ADF, Azure ADLS
Scripting
Python
Operating Systems
Linux, Unix and Windows
IDEs
Eclipse, Intellij, PyCharm
Database
MySQL, HBase
Projects Undertaken at Deltacubes Technology Pvt. Ltd.
Project Name: Depletions
Duration: 3 Year(April 2019- Presents)
Description: Diageo is a worldwide liquor manufacturer and distributor. Data lake team helps the Diageo Business to get the data from all customer/distributor across the world.
So Data Lake team collects the data from different Sources then cleansing, standardizes and harmonizes to get meaningful data for business analytics team.
Responsibilities:-
1.) Cleansing and Standardizing Raw data from multiple file formats (xlsx ,csv, JSON) using Spark/Scala.
2.) Generating Parquet/CSV files after Harmonizing Data for Business Analytics Team
3) Generating CSV files for Anaplan Team with EU Calculations in Blob Storage
4) Generating Parquet files for Sellout Team
5.) Orchestrating Data pipelines using ADF.
6) Creating Views on Dremio on top of ADLS.
7) Parsing JSON for Standardizing Raw Data to get the Rules.
8) Trigger the Pipeline from ADF.
9) Created CI/CD pipeline for ARM Templates.
Technology: Spark, Scala, Dremio, Azure(ADF,ADLS, Blob), Databricks
Projects Undertaken at Tavant Technologies
Project Name: Experian BIS SALT
Duration: 2 Years(Jan 2017- Apr 2019)
Description:
Experian is a consumer credit reporting agency. Experian collects and aggregates information on and over one billion people and businesses. It is one of the ‘Big Three’ credit reporting agencies.
This project is to import the SBFE data that Experian got recently into Big Data system and make it available to internal/external customers for analysis and credit score modelling for any business.
Responsibilities:-
1.) Develop Spark/Scala code for validating Semi Structured Data and loading data to system.
2.) Generate Avro file from CSV file for integrating with external System(One Search) using Spark/Scala.
3.) Create Hive tables for Data Management team.
4.) Make data available for Commercial Data Sciences team for analyzing in SAS.
5.) Run validation job on historical data and make data ready for using.
6.) Support functional testing and Bug Fixing in Spark code.
7.) Write Spark Data Frame code to implement product view rule on processed data.
8.) Write Spark Data Frame to read and analyze nested complex Avro data.
Technology: Spark, Scala, HBase, Hive
Projects Undertaken at AMD India Pvt. Ltd(Contingent Worker Through Magna Infotech)
Project Name: Scan-view
Duration: 11 Months(Feb 20
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
