Latent View Analytics

Lead Data Engineer                                                                                         Chennai, IN | Sep '21 - Present

Project: Fiserv (MSMO)

Objective:MSO is the anomaly detection project; we use to move the data from SQL server to Hadoop and do the transformation activities and create the final transformed table which can help the Data science team to create an effective model to get insight out of it.

·        Developed Spark programs using Pythonto execute the data quality rules on top of the Hive tables.

·        Performing data processing using PySpark when we get the data from other sources

.

·        Performed Data enrichments such as filtering, pivoting, format modeling, sorting and aggregation using Hive.

·        Created DAGs to automate the ETL/ELT process inAirflow.