Job Description

KEY ACCOUNTABILITIES

• Data Quality Policy & Framework Implementation

• Define and operationalize enterprise Data Quality policies, procedures, and standards.

• Establish standardized data quality dimensions and certification frameworks.

• Implement scalable validation frameworks across ingestion, transformation, and serving layers.

• Embed “quality-by-design” principles into data product lifecycle.

• Data Observability Platform Development

• Design and implement end-to-end data observability capabilities including:

  • Data freshness and SLA monitoring

  • Volume and distribution anomaly detection

  • Schema drift and pipeline health monitoring

  • Data lineage validation and reliability tracking

  • Develop automated alerting and incident detection mechanisms.

• Custom Data Applications (DataApps) Development

• Build custom Data Quality and Observability applications using:

  • Databricks native capabilities

  • Streamlit / Databricks Apps

  • Python-based backend services

• Develop user interfaces enabling:

  • Data quality rule configuration

  • Dataset certification workflows

  • Quality score visualization

  • Issue tracking and remediation workflows

  • Enable self-service quality monitoring for engineering and analytics teams.

• Azure & Databricks Platform Integration

• Implement data quality checks within Azure-based data pipelines and Databricks workflows.

• Integrate monitoring with:

  • ADLS Gen2

  • Databricks Lakehouse architecture

  • Batch and streaming pipelines

  • Develop reusable frameworks leveraging Spark and Delta Lake.

  • Optimize performance and scalability of quality validation workloads.

• Automation & Engineering Excellence

• Integrate DQ checks into CI/CD and deployment pipelines.

• Develop metadata-driven quality monitoring solutions.

• Implement automated remediation and self-healing workflows where applicable.

• Ensure auditability, traceability, and governance compliance.

• Metrics, Reporting & Adoption

• Define enterprise Data Quality KPIs and reliability SLAs.

• Build dashboards tracking platform-wide data trust scores.

• Drive adoption of standardized DQ practices across engineering teams.

• Support audit and compliance reporting initiatives.

• Data Quality Score

• Leadership & Collaboration

• Act as technical lead for Data Quality and Observability engineering.

• Mentor engineers on best practices for data reliability.

• Collaborate with Data Engineering, Governance, and Platform Architecture teams.

• Contribute to long-term evolution of the enterprise data platform.

QUALIFICATIONS, EXPERIENCE AND SKILLS

Education

• Bachelor’s or master’s degree in computer science, Data Engineering, Information Systems, or related field.

Experience

• 8+ years of experience in Data Quality engineering roles within Data Platforms/Data Engineering teams.

• Proven experience building custom applications on Databricks or data platforms.

• Experience designing enterprise Data Quality or Data Observability solutions.

• Hands-on experience developing internal data tools or platform applications.

Technical Skills (Required)

• Cloud & Data Platform

  Strong expertise in:

• Microsoft Azure
• Databricks Lakehouse platform
• ADLS Gen2
• Distributed data processing using Spark
• Application & DataApp Development
  • Experience building DataApps using:
    • Streamlit
    • Databricks Apps or notebook-based applications
    • Python backend development
    • Experience designing UI-driven data engineering tools or internal platforms.
  • Data Quality & Observability
    • Experience implementing data validation frameworks.
    • Strong SQL and Python programming skills.
    • Knowledge of anomaly detection, monitoring, and data reliability concepts.
  • Engineering & Integration
    • CI/CD integration for data pipelines.
    • REST API integrations and automation workflows.
    • Metadata-driven architectures and lineage concepts.

Core Competencies

• Platform-first engineering mindset.
• Strong problem-solving and analytical thinking.
• Ability to translate governance requirements into scalable technical solutions.
• Strong stakeholder collaboration and communication skills.
• Ownership mindset with ability to lead initiatives end-to-end.

Preferred

• Experience with Great Expectations, Deequ, Soda, or similar frameworks.
• Experience with streaming data validation.
• Exposure to AI-driven data observability or anomaly detection.
• Experience building enterprise internal developer platforms.

#LI-AA6