Sr Data Engineer
Location : Jaipur, India
Experience : 8 Year
Rate: $22 / Hourly
Availability : Immediate
Work From : Offsite
Category : Information Technology & Services
01/2021–present SeniorLead DataEngineer
InvolvedinBusinessrequirementdocumentwalkthroughandclientcallstounderstandfunctionality
DesignedastablearchitectureforProject
Involved in planning and development of ETL pipelines using ADFUnblockingtheteammembersinrequirementsandtechnicalissuesDesignedAzureDataFactoryPipelinestomovedatatoAzureDataLakeImplementedADFPipelinestopulldatafromdifferentsources
Accessed Azure Data Lake files data in Azure Data bricks using mount pointsImplementedBusinesslogicsinSparkSQL/PysparktopopulaterequiredtablesHandledcleansing,aggregationsandtransformationsinSpark
ImplementedAzureFunctionswithPythontopulldatafromAPIsandscheduledtheseAzureFunctionsinAzureDataFactory
ImplementedendtoendloggingframeworkforAzureDatafactorypipelineDesignedofDatawarehouseforstableimplementation.
IntegratedAzureDataFactoryforschedulingJobsandTriggeringotherrelatedJob.ExtensivelyusedAzureKeyVaultstoconfiguretheconnectionsinLinkedservicesCreatedLogicAppstotriggerthemailnotificationfromADFPipelines.
ImplementedStoredProceduresbusinesslogicinAzureSQLDatawarehouse.
CreatedCI/CDpipelinesforADF,AzureSQLDatawarehouseandAzureSQLDatabasedeploymentinADO
DeployedPipelineandcodetomultipleenvironmentsusingCI/CDpipelineValidatedthedataatoverallandeverydimensionlevel.
WrittentestcasestovalidatedatainADO
09/2018–12/2020 SeniorSoftwareEngineer
DesignedofDatawarehouseforstableimplementation.
Designed anddevelopedAzureDatafactorypipelines forDatamigration.IntegratedAzureDataFactoryforschedulingJobsandTriggeringotherrelatedJob.
Created LogicApps topull thedata fromSharePoint excelsto ADLS.ImplementedAzureDatabricksnotebooksbusinesslogicatDatawarehouselevel.CreatedamountpointinPythontoaccessthefilesresideinADLS.
CreateduserdefinedfunctionsusingPython.
Createddataframes,viewsandtablesusingPythoninAzureDatabricks.
Used Python loops and aggregation techniques to process data in Azure Databricks.PublishedtheFactandDimensiontablesfromADBtoAzureSQLDatawarehouseandADLSusingPython.
CreatedvalidationframeworkusingPythoninAzureDatabricks.
CreatedTabularmodelonFactandDimensionviewswhichareinAzureSQLDatawarehouse.
UpdatedTabularlevelmeasuresanddimensions,wroteDAXqueriesforcalculatedmeasures.
ValidatedthedataatoverallandeverydimensionlevelinTabular,checkedondataisslicingateverydimensionlevel
WrittentestcasestovalidatedatainatTabularinADODevelopedreportsusingPowerBI.
07/2017–08/2018 SoftwareEngineer2
Designedanddevelopedrelationaldatabases.Guidingteammembers.
DevelopedSSISpackagestoloadthedatafromdifferentsourcestoSQLandcreateaDatawarehouse.
ImplementedStoredProcedurestoprocessbusinesslogicatDatawarehouselevel.CreatedTabularmodelontopofDatawarehouse
ImplementedmainbusinesslogicforfinancialmetricsinTabularusingDAXqueriesValidatedthedataatoverallandeverydimensionlevelinTabular,checkedondataisslicingateverydimensionlevel
DevelopedreportsusingPowerBI.
WroteautomatedBVTSPstocheckvaluesateachlevelonrefreshofdatadevelopedscriptfilesonbigDataplatformtomaketheheavydatatobeprocessed
Involved in Business requirement document walkthrough and client calls to understand functionality
Designed a stable architecture for Project
Involved in planning and development of ETL pipelines using ADF Unblocking the team members in requirements and technical issues Designed Azure Data Factory Pipelines to move data to Azure Data Lake Implemented ADF Pipelines to pull data from different sources
Accessed Azure Data Lake files data in Azure Data bricks using mount points Implemented Business logics in Spark SQL/Pyspark to populate required tables Handled cleansing, aggregations and transformations in Spark
Implemented Azure Functions with Python to pull data from APIs and scheduled these Azure Functions in Azure Data Factory
Implemented end to end logging framework for Azure Data factory pipeline Designed of Data warehouse for stable implementation.
Integrated Azure Data Factory for scheduling Jobs and Triggering other related Job. Extensively used Azure Key Vaults to configure the connections in Linked services Created Logic Apps to trigger the mail notification from ADF Pipelines.
Implemented Stored Procedures business logic in Azure SQL Data warehouse.
Created CI/CD pipelines for ADF, Azure SQL Datawarehouse and Azure SQL Database deployment in ADO
Deployed Pipeline and code to multiple environments using CI/CD pipeline Validated the data at overall and every dimension level.
Written test cases to validate data in ADO
.
{{contact.cdata.phone}}
{{contact.cdata.email}}
View Resume
