Piyush

Senior Data Engineer

● Helped and Managed to create Central Data Lake with various integrations for different businesses.

● Created and managed real time pipeline that is using Confluent Kafka , Spark Structured Streaming and Confluent Kafka Connectors for Customers Transaction data at very large scale.

● Identified bottlenecks of existing Spark Jobs and redesigned to cut resources and time.

● Designed and implemented Monitoring Framework for all the pipelines triggered using Airflow.

● Helped Multiple Teams in same competency for Pipelines and design of the system.

● Created Data models in Vertica and Postgres that can support dynamic queries of Power BI for Analytics reports.

● Created REST API in Scala with http4s and Hikari libraries. Created Generic pipelines to transform and load from Vertica DB to Postgres and S3 using Spark in AWS. Innoeye Technologies 2020-05 to 2021-07 Senior Data Engineer

● Collaborated with multiple teams and management to understand the requirement and design the whole system.

● Optimized and re-architected Spark Jobs to cut the resources by 80%.

● Responsible for creating Spark Jobs using Spark SQL and Java for Business Intelligence applications. •

● Developed NiFi Workflows to schedule spark jobs and manage ingestion of data.

● Built data models in HBase and MySQL for web applications and big data analytics .

● Used spring-boot and Hibernate to write REST APIs in microservices architecture.

● Built Streaming services for real time processing of 10M users using Java and Kafka