Job Description

How you'll make an impact

• Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products.
• Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD.
• Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers.
• Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team.
• Own your pipelines in production — performance, cost, reliability, and incident response.
• Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform.

What you'll need (Required)

• Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks.
• Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns.
• Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments.
• Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting.
• Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL.
• Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments.
• Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions.
• "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication."
• "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads."

What else we look for (Preferred)

• Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns.
• Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments.
• Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting.
• Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL.
• Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments.
• Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code.
• Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication.
• Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads.
• Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow.
• Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse.
• Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment.
• Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.