APRIL 2018 – PRESENT
Project: TACC, MARS, CTM - Oil & Gas Facilities. Client: Chevron
Designation: Team Lead
· Worked on Conversion of IBM DataStage ETL tool jobs to Azure Data Factory which involves Understanding of DataStage graphical framework for developing the jobs to move data from source systems to target systems.
· Created Azure ETL Accelerators framework – Created Audit and Control tables in Azure SQL for Data Extraction, Change data capture and Data load. These tables are also used for storing job statistics (Start time, End time, Status (Success/Failure/Started), Record Counts (source, target/reject) and Elapsed Time) and are updated frequently during job execution.
· Worked on implementation of complex HR processes like Employee workflow, SAP BANK workflow, Workday Bank workflow, Timesheets, Employee promotion and Employee termination processes.
· Perform default Pagination in REST API data using Absolute URLs and manual pagination if continuation token is in request body using until activities.
· Read the data from REST API and write to Data Lake/Blob Storage in Json file and flatten out the nested complex Json into csv file using Azure Data Flows.
· Created linked services, Datasets and Pipelines for data movement from Source to Staging, Staging to Integration and Integration to Data ware house DB and Created new pipelines for different types of databases like Oracle On-premises to Azure SQL, SQL On-premises to Azure SQL and Azure SQL to Azure SQL.