Data Engineer

Remote, India
Posted on 18 Oct, 2021
406 views
4 applications
ContractRemote/OnsiteVerified

Job Requirements

Experience

2 - 5 years

No. of Positions

4

Duration

3-6 months

Skills

pysparkPythonsparkhadoophiveHDFS

Job Description

Skills Required: PySpark, Python, SQL, Spark SQL, HIVE, HDFS, Spark


Roles and Responsibilities:

  • Responsible for developing and maintaining applications with PySpark
  • Contribute to the overall design and architecture of the application developed and deployed.
  • Performance Tuning wrt to executor sizing and other environmental parameters, code optimization, partitions tuning, etc
  • Interact with business users to understand requirements and troubleshoot issues.
  • Implement Projects based on functional specifications.

Must-Have Skills:

  • Good experience in Pyspark - Including Dataframe core functions and Spark SQL
  • Good experience in SQL DBs - Be able to write queries including fair complexity.
  • Should have excellent experience in Big Data programming for data transformation and aggregations
  • Good at ETL architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
  • Having exposure to SCALA and Apache Airflow is desirable
  • Good customer communication.
  • Good Analytical skills


Interested in this job?

Job Overview

  • Location

    Remote, India

  • Job Type

    Contract

  • Rate

    $ 14 / Hourly

  • Work From

    Remote/Onsite

  • Start Date

    24 Oct, 2021