Data Brick Engineer

We are seeking a skilled Data Engineer with hands-on experience in Databricks, Python, Spark Streaming, SQL, and PySpark to join our data engineering team. The ideal candidate will be responsible for designing, developing, and optimizing data processing pipelines and solutions, leveraging Databricks and Apache Spark technologies.

Key Responsibilities:

• Data Engineering and Processing:

  • Design, develop, and maintain scalable data pipelines and workflows using Databricks, Spark Streaming, and PySpark.
  • Implement and optimize real-time data processing solutions to handle large volumes of data with low latency.

• Databricks and Spark:

  • Leverage Databricks for data processing, analytics, and machine learning tasks.
  • Develop and manage Spark applications using PySpark for data transformation and analysis.

• Data Integration and Management:

  • Integrate data from various sources, including databases, APIs, and streaming platforms, into Databricks.
  • Ensure data quality and consistency across all data processing systems and pipelines.

• SQL and Data Analysis:

  • Write and optimize complex SQL queries for data extraction, transformation, and analysis.
  • Use SQL to interact with relational databases and Databricks SQL Analytics for reporting and visualization.