Job Description

Job Summary

Azure Databricks Platform Support Engineer

Job Description

Role Overview

We are seeking an Azure Databricks Platform Support Engineer to ensure the reliability, performance, and security of our Databricks environment. The role focuses on day-to-day platform administration, production troubleshooting, and enabling data engineering and analytics teams across the organisation.

Key Responsibilities

• Administer and support Azure Databricks workspaces, compute, jobs, notebooks, and SQL Warehouses.
• Troubleshoot and resolve production incidents by analysing job logs, cluster events, error messages, and audit information.
• Manage Unity Catalog governance and ADLS Gen2 integration, including permissions and access controls.
• Configure and maintain compute policies, instance pools, autoscaling, auto-termination, and cost controls.
• Support workspace networking and connectivity, including VNet injection, private endpoints, and Private Link.
• Maintain operational documentation, runbooks, and change records for the platform.

Required Skills

• Hands-on experience with Azure Databricks platform support or administration.
• Good understanding of Databricks workspaces, compute, jobs, notebooks, SQL Warehouses, libraries, and permissions.
• Experience troubleshooting Databricks production issues.
• Working knowledge of Unity Catalog and ADLS Gen2 integration.
• Understanding of Azure networking basics: VNet, subnet, NSG, route table, NAT Gateway, firewall, private endpoint, Private Link, and DNS.
• Familiarity with compute policies, instance pools, runtime versions, autoscaling, auto-termination, and cost controls.
• Ability to read job logs, cluster events, error messages, and audit information.
• Strong communication and documentation skills.

Preferred Skills

• Experience with VNet-injected Databricks workspaces and Secure Cluster Connectivity.
• Experience with Azure Key Vault, Azure Monitor, Log Analytics, and diagnostic logs.
• Knowledge of PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• Experience with Databricks CLI, REST APIs, Terraform, Bicep, Azure DevOps, or GitHub.
• ITIL-style incident, problem, and change management experience.

Tools & Technologies

Azure Databricks · Unity Catalog · Delta Lake · Apache Spark / PySpark / Spark SQL · ADLS Gen2 · Azure Key Vault · Azure Monitor / Log Analytics · Azure Networking · Databricks CLI / REST APIs · Azure DevOps / GitHub

Key Responsibilities

1. Lead the design, development, and deployment of snowflake, azure data factory (adf), and data bricks solutions according to project requirements.
2. Collaborate with cross functional teams to gather data requirements, design data pipelines, and ensure data quality and integrity.
3. Provide technical guidance and mentorship to team members on best practices for utilizing snowflake, azure data factory (adf), and data bricks.
4. Troubleshoot and resolve technical issues related to data pipelines, data transformation, and data storage in snowflake, azure data factory (adf), and data bricks.
5. Stay updated on industry trends and best practices related to snowflake, azure data factory (adf), and data bricks to recommend and implement improvements in existing data processes.

Skill Requirements

1. Proficiency in snowflake architecture, implementation, and optimization.
2. Handson experience with azure data factory (adf) including data pipeline development, integration, and monitoring.
3. Strong knowledge of data bricks for big data processing, data engineering, and machine learning workflows.
4. In-depth understanding of data warehousing concepts, etl processes, and data modeling techniques.
5. Excellent problem-solving skills and ability to work in a fast paced environment.
6. Strong communication and leadership skills to effectively collaborate with team members and stakeholders.

Other Requirements

null