Job Description

• Build scalable data ingestion pipelines for relational systems and unstructured/semi-structured sources.
• Design and optimize lakehouse workloads using Apache Iceberg
• Optimize table layouts, partitioning strategies, compaction, and schema evolution for Iceberg-based datasets.
• Implement best practices for versioning, time travel, incremental processing, and ACID guarantees
• Develop and tune Apache Spark (batch & streaming) for performant, scalable data processing.
• Work with AWS services (Glue, EMR, Lambda, Step Functions, S3) focusing on cost optimization and performance tuning
• Implement and manage Kafka and Kafka streaming pipelines for large-scale real-time processing.
• Build and orchestrate data workflows using DBT and Airflow
• Leverage AI/LLM automation to accelerate ETL/ELT development, validation, and debugging.
• Implement observability—monitoring, logging, and alerting—to ensure platform stability and reliability.
• Participate in code reviews to ensure code quality and share best practices with the team.

Required Skills

• Strong SQL, Python, and Spark experience
• Deep experience ingesting and processing structured & unstructured data
• Strong knowledge of Apache Iceberg and lakehouse architectures
• Hands-on AWS experience with tuning and cost optimization
• Expertise with Kafka, Kafka Streaming, DBT, and Airflow
• Solid understanding of distributed systems and data engineering best practices