Job Description

We are looking for an experienced Data BAU Lead – Data Engineering to lead data operations, production support, and continuous improvement initiatives within a cloud-first data environment. The ideal candidate will have strong technical leadership capabilities, hands-on expertise in AWS, Snowflake, Data Engineering, and BAU operations, with proven experience managing large-scale data platforms, production incidents, and data transformation initiatives.

The role requires a strong understanding of modern data architecture, cloud technologies, ETL pipelines, data lakes, data warehouses, and the ability to lead engineering teams while collaborating with business stakeholders, architects, data scientists, and DevOps teams.

Key Responsibilities

Data Engineering & Platform Management

• Lead the Data Engineering BAU team responsible for daily operations, production support, incident management, and platform stability.
• Design, develop, and maintain scalable data pipelines and data processing solutions.
• Develop tools and frameworks to improve data movement between internal/external systems and enterprise data platforms.
• Build robust and reusable data ingestion pipelines to collect, cleanse, transform, harmonize, and consolidate data from multiple sources.
• Support and enhance existing data applications, infrastructure, and architecture.
• Develop and maintain datasets, data models, and data management processes.
• Improve data quality, reliability, performance, and operational efficiency.

Cloud & Data Platform Migration

• Support migration of existing data transformation workloads from Oracle and MS SQL environments to Snowflake
• Lead migration of legacy transformation processes from Oracle, Hive, and Impala into modern cloud-based solutions using Spark, Python, and AWS Glue
• Design and optimize cloud-based data solutions using AWS services.
• Evaluate and recommend data technologies, platforms, and tools aligned with business strategy.

Production Support & BAU Operations

• Manage critical production support activities, ensuring timely resolution of incidents and operational issues.
• Establish and improve BAU processes, monitoring, alerting, and operational controls.
• Collaborate with DevOps teams to enhance CI/CD deployment processes and system monitoring.
• Ensure proper documentation of processes, workflows, technical solutions, and operational procedures.

Stakeholder & Team Leadership

• Work closely with business stakeholders to understand data requirements, availability, scalability, and accessibility needs.
• Lead requirement analysis and deliver effective data solutions.
• Collaborate with Data Scientists, Solution Architects, Engineers, and Business Teams on analytics initiatives.
• Mentor and guide data engineers, promoting coding best practices, design principles, and engineering excellence.
• Manage stakeholder communication including senior leadership updates when required.

Required Technical Skills

Cloud & Data Platforms

• Strong hands-on experience with AWS Cloud Data Services, including:
• Amazon S3
• AWS Glue
• AWS DMS
• AWS MWAA (Managed Workflows for Apache Airflow)
• IAM
• Amazon RDS
• Amazon Kinesis
• AWS Lambda
• AWS Step Functions
• Strong understanding of cloud architecture design and optimization techniques.
• Extensive knowledge of Snowflake architecture, performance optimization, and implementation practices
• Experience working with enterprise-scale Data Lakes and Data Warehouses.

Data Engineering Skills

• Strong programming experience in:
• SQL
• Python
• PySpark
• Unix/Linux Shell Scripting
• Strong experience with:
• ETL/ELT development
• Data ingestion frameworks
• Data integration
• Data transformation
• Data modelling
• Data quality management
• Good understanding of distributed systems and scalable data processing.
• Experience with data streaming technologies is preferred.

Workflow & DevOps

• Strong knowledge of:
• Apache Airflow / AWS MWAA
• CI/CD pipelines
• Git-based deployments
• Agile development methodologies
• SDLC lifecycle

Experience & Qualification Requirements

• 8+ years of experience in Data Engineering.
• 5+ years of recent hands-on coding experience as a Lead Engineer managing production support and BAU operations.
• Minimum 2+ years of experience with large-scale datasets, Data Lakes, and Data Warehouse technologies
• Strong preference for hands-on experience with Snowflake
• Minimum 3+ years of experience with AWS Data Engineering services including AWS Glue, S3, RDS, Lambda, Kinesis, Step Functions, and Airflow.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or related STEM discipline.
• Experience working in Agile, dynamic, customer-focused environments.