Job Description

Data Engineer – Responsibilities (4–7 Years Exp)

• Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala.
• Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing.
• Design and implement data models including Dimension and Fact tables, with clearly defined grain, relationships, and measures.
• Implement Slowly Changing Dimensions (SCD) and appropriate incremental processing strategies.
• Work with AWS data engineering and serverless services such as Glue, S3, Lambda, EMR, and Step Functions.
• Develop and maintain reliable, scalable data pipelines with appropriate error handling, logging, monitoring, and data quality checks.
• Apply data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
• Work with Apache Iceberg, including table design, partitioning, schema evolution, and incremental processing.
• Develop data transformation workflows using DBT and orchestrate pipelines using Airflow.
• Troubleshoot and optimize data pipelines for performance, scalability, and cost efficiency.
• Participate in code reviews, testing, debugging, and continuous improvement of data engineering solutions.

Required Skills

• 4–7 years of relevant Data Engineering experience.
• Strong SQL skills — SQL is a core requirement.
• Strong hands-on experience with Apache Spark and Python or Scala.
• Hands-on experience with AWS data engineering and serverless services, particularly AWS Glue and S3.
• Good understanding of Data Modelling, including:
• Dimension and Fact table design
• Defining appropriate Fact grain
• Dimension-to-Fact relationships
• Implementing SCD Type 1 / Type 2 and other appropriate SCD patterns
• Strong understanding of data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
• Good understanding of distributed data processing and Spark performance optimization.

Additional Responsibilities / Good to Have

• Experience with Apache Iceberg and lakehouse architectures.
• Experience with DBT for data transformation and modelling.
• Experience with Airflow for workflow orchestration.
• Kafka / Kafka Streaming experience is a good to have.
• Exposure to AI/LLM tools for development, debugging, testing, and documentation.