Job Description

Principal Engineer, Data Platform (Databricks)

About the Role

We are building our next-generation data platform on Databricks. The Principal Engineer sets the technical direction for how the bank ingests, models, governs and serves data. This is a hands-on role: you will design the detailed lakehouse implementation, set engineering standards for SQL and pipelines, lead the migration from legacy platforms, and mentor engineers across several teams. You will work within the target-state architecture owned by the Solution Architect.

Key Responsibilities

• Lakehouse design: Own the detailed design of the Databricks lakehouse: ingestion, bronze/silver/gold layers, Delta Lake design, Unity Catalog setup, and workspace and cluster configuration.
• Migration: Lead the hands-on migration of legacy Hadoop/Hive/Spark and warehouse workloads to Databricks, including re-platforming HQL and SQL, validating data parity, and safe cutover.
• Engineering standards: Set standards for SQL, PySpark and Delta Live Tables/Lakeflow pipelines, covering code review, testing, CI/CD and observability.
• Performance and cost: Tune performance and cost: partitioning, liquid clustering/Z-ordering, file compaction, join and shuffle tuning, Photon, cluster right-sizing and DBU spend governance.
• Data modeling: Design canonical, auditable entity models (applications, accounts, customers, transactions) across source systems.
• Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
• Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
• Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
• Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
• Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.

Must-Have Skills

• Experience: 12+ years in data or software engineering, with at least 5 years in a technical lead, staff or principal role.
• SQL: Expert-level SQL: complex analytical queries, window functions, CTEs, query plan analysis and tuning on very large datasets (billions of rows).
• Spark and big data: Deep experience with Apache Spark (PySpark and/or Scala) and the ecosystem: Hadoop, Hive, Kafka, Parquet/ORC/Delta.
• Databricks in production: Hands-on production use of Delta Lake, Unity Catalog, Workflows/Jobs, Databricks SQL, and cluster and cost management.
• Pipelines: Proven design and operation of large-scale batch and streaming pipelines, including Structured Streaming, Delta Live Tables and Kafka, on a major cloud.
• Migration: Experience migrating on-premises or EMR-based Hadoop/Hive platforms to Databricks.
• Performance and cost: Proven Spark and Databricks performance tuning and cost optimization, including cluster policies.
• Data modeling: Dimensional, entity-centric, Data Vault or medallion modeling, including slowly changing dimensions, incremental/CDC loads and soft deletes.
• Orchestration and CI/CD: Airflow, Databricks Workflows or Oozie; Git, Terraform or Databricks Asset Bundles, and automated testing and data quality checks.
• Data security: Implementation of PII protection: encryption, tokenization, and row/column-level security.
• Leadership and communication: Mentoring engineers, and explaining technical trade-offs clearly to engineers and executives.

Good-to-Have Skills

• Banking, lending or other regulated financial services experience
• Data modeling for Marketing (campaign, attribution, funnel) and Collections (delinquency, contact, payments)
• Observability and monitoring frameworks for data pipelines
• Databricks Certified Data Engineer Professional
• Lakehouse design: Own the detailed design of the Databricks lakehouse: ingestion, bronze/silver/gold layers, Delta Lake design, Unity Catalog setup, and workspace and cluster configuration.
• Migration: Lead the hands-on migration of legacy Hadoop/Hive/Spark and warehouse workloads to Databricks, including re-platforming HQL and SQL, validating data parity, and safe cutover.
• Engineering standards: Set standards for SQL, PySpark and Delta Live Tables/Lakeflow pipelines, covering code review, testing, CI/CD and observability.
• Performance and cost: Tune performance and cost: partitioning, liquid clustering/Z-ordering, file compaction, join and shuffle tuning, Photon, cluster right-sizing and DBU spend governance.
• Data modeling: Design canonical, auditable entity models (applications, accounts, customers, transactions) across source systems.
• Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
• Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
• Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
• Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
• Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.

Graduate in Computer Science, Data Science, or a related field. 12+ years of experience in data modelling or data analysis.