The Senior Data Engineer is responsible for the development of high-performance, distributed computing solutions using technologies like Apache Spark, Apache Flink/Storm/Kafka Streams, Kafka, NoSQL database, and other distributed systems based on the needs of the organization.
Bachelor's Degree or Master's Degree in Computer Science or related field with 8-10 years of experience.
Expertise in handling huge datasets with deep experience in AWS, Spark, Flink, DynamoDB, Cassandra, NoSQL, etc.
Experience with data pipeline orchestrating tools: Airflow, Luigi, etc.
Experience with container orchestration tools like Kubernetes. (Must have)
Experience with AWS cloud services: S3, EMR, RDS, Redshift, Lambda.
Strong programming skills in at least one of the following: Java, Scala.
Experience in building scalable complex data pipelines in the production system.
Understanding of stream processing with good knowledge of Kafka and either Kafka Streams/Spark streaming/Storm or Flink.
Knowledge of software engineering best practices CI/CD, log aggregation/monitoring/alerting for a production system.