Job Description

Responsibilities

Data Pipeline Development & Management

• Design, build, and maintain scalable data pipelines using AWS Glue
• Implement ETL/ELT processes to ingest data from multiple sources
• Optimise data workflows for performance, reliability, and scalability
• Monitor and troubleshoot pipeline failures and performance issues

Data Infrastructure & Engineering

• Manage and optimise AWS Redshift data warehouse operations
• Configure and maintain data storage solutions, including AWS S3 and data lakes
• Implement data partitioning, indexing, and compression strategies
• Support Infrastructure as Code (IaC) for data platform deployment

CI/CD & DevOps for Data

• Develop and maintain CI/CD pipelines for data workflows using GitLab
• Implement automated testing for data pipelines and data quality
• Manage version control and deployment strategies for data assets
• Configure AWS Lambda functions for data processing and automation

Monitoring & Support

• Set up monitoring and alerting for data pipeline health and performance
• Provide technical support for data-related issues
• Collaborate with technical teams on data architecture and enhancements
• Optimise query performance and database operations

Documentation & Reporting

• Document data pipeline architecture and technical specifications
• Maintain runbooks and operational procedures
• Provide regular updates on system health and progress
• Track engineering tasks using Jira and maintain documentation in Confluence

Requirements

• Strong experience in data engineering and data pipeline development
• Proficient in SQL, Python, and shell scripting
• Hands-on experience with AWS data services such as Redshift, S3, Glue, Lambda, and CloudWatch
• Experience in data warehouse design and performance optimisation
• Strong knowledge of CI/CD pipelines (GitLab preferred)
• Experience with Infrastructure as Code tools such as Terraform or CloudFormation
• Solid understanding of data modelling and database design principles
• Strong troubleshooting and performance tuning skills