OnBenchMark Logo

Piyush

Senior Data Engineer
placeholder
Location Neemuch
view-icon
Total Views65
availability
Shortlist0
back in time
Member since30+ Days ago
 back in time
Contact Details
phone call {{contact.cdata.phone}}
phone call {{contact.cdata.email}}
Candidate Information
  • User Experience
    Experience 6 Year
  • Cost
    Hourly Rate$9
  • availability
    AvailabilityImmediate
  • work from
    Work FromOffsite
  • check list
    CategoryEngineering & Design
  • back in time
    Last Active OnJune 02, 2024
Key Skills
Data WarehousingETLPythonJavaMicroservices
Summary

Senior Data Engineer

● Helped and Managed to create Central Data Lake with various integrations for different businesses.

● Created and managed real time pipeline that is using Confluent Kafka , Spark Structured Streaming and Confluent Kafka Connectors for Customers Transaction data at very large scale.

● Identified bottlenecks of existing Spark Jobs and redesigned to cut resources and time.

● Designed and implemented Monitoring Framework for all the pipelines triggered using Airflow.

● Helped Multiple Teams in same competency for Pipelines and design of the system.

● Created Data models in Vertica and Postgres that can support dynamic queries of Power BI for Analytics reports.

● Created REST API in Scala with http4s and Hikari libraries. Created Generic pipelines to transform and load from Vertica DB to Postgres and S3 using Spark in AWS. Innoeye Technologies 2020-05 to 2021-07 Senior Data Engineer

● Collaborated with multiple teams and management to understand the requirement and design the whole system.

● Optimized and re-architected Spark Jobs to cut the resources by 80%.

● Responsible for creating Spark Jobs using Spark SQL and Java for Business Intelligence applications. •

● Developed NiFi Workflows to schedule spark jobs and manage ingestion of data.

● Built data models in HBase and MySQL for web applications and big data analytics .

● Used spring-boot and Hibernate to write REST APIs in microservices architecture.

● Built Streaming services for real time processing of 10M users using Java and Kafka

Copyright© Cosette Network Private Limited All Rights Reserved
Submit Query
WhatsApp Icon