Job Description

Job Summary

• Develop automated scripts to validate source-to-target data movements, ensuring schema consistency, completeness, and timeliness.
• Build robust, reusable testing utilities using Python, native AWS services that integrate seamlessly into DataOps workflows.
• Set up continuous data drift monitoring, schema validation, and data quality dashboards to proactively catch pipeline failures.
• Partner with data engineers, BI teams, and software engineers to translate business rules into strict data quality KPIs and SLAs.

Key Responsibilities

• Develop automated scripts to validate source-to-target data movements, ensuring schema consistency, completeness, and timeliness.
• Build robust, reusable testing utilities using Python, native AWS services that integrate seamlessly into DataOps workflows.
• Set up continuous data drift monitoring, schema validation, and data quality dashboards to proactively catch pipeline failures.
• Partner with data engineers, BI teams, and software engineers to translate business rules into strict data quality KPIs and SLAs.

Skill Requirements

• Deep proficiency in AWS data and storage services like Amazon S3, AWS Glue, Amazon Redshift, and AWS Lambda.
• Advanced proficiency in Python for building automation frameworks and orchestrating data tests.
• Experience with orchestration tools (e.g., Apache Airflow), data transformation, and data quality monitoring frameworks (e.g., Amazon Deequ).
• Strong command of SQL and UNIX/Linux scripting for querying and analyzing large-scale, distributed datasets

Certifications: Candidate with AWS Certified Data Engineer – Associate - Highly preferred

Other Requirements

null