Consultant (Data Engineer)

Objective

Data Engineer will be responsible for expanding and optimizing our data and database architecture,

as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an

experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building.

The Data Engineer will support our software developers, database architects, data analysts and data

scientists on data initiatives and will ensure optimal data delivery architecture is consistent

throughout ongoing projects. They must be self-directed and comfortable supporting the data needs

of multiple teams, systems, and products

Roles and Responsibilities:

➢ Good knowledge on GCP Data Flow, Big Query, Google cloud storage, cloud function, Cloud

composer, etc.

➢ Good knowledge on working on the access permissions like IAM.

➢ Understanding the upstream data designing efficient data pipelines, building data marts in

Big Query, having the transformed data exposed to downstream applications.

➢ Should be comfortable in building and optimizing performant data pipelines which include

data ingestion, data cleansing and curation into a data warehouse, database, or any other

data platform using PySpark.

➢ Experience in distributed computing environment and Spark architecture.

➢ Optimize performance for data access requirements by choosing the appropriate file

formats (AVRO, Parquet, ORC etc.) and compression codec respectively.

➢ Experience in writing production ready code in Python and test, participate in code reviews

to maintain and improve code quality, stability, and supportability.

➢ Experience in designing data warehouse/data mart.

➢ Experience with any RDBMS preferably SQL Server and must be able to write complex SQL

queries.

➢ Expertise in requirement gathering, technical design and functional documents.

➢ Experience in Agile/Scrum practices.

➢ Experience in leading other developers and guiding them technically.

➢ Experience in deploying data pipelines using automated CI/CD approach.

➢ Ability to write modularized reusable code components.

➢ Proficient in identifying data issues and anomalies during analysis.

➢ Strong analytical and logical skills.

➢ Must be able to comfortably tackle new challenges and learn.

➢ Must have strong verbal and written communication skills.