Job Description

Job Summary

We are looking for a highly skilled Databricks Engineer to design, build, optimize, and maintain large-scale data processing pipelines and analytics workloads on the Databricks platform. This role requires strong hands-on experience with Databricks Jobs, Delta Tables, Pipelines, Spark SQL, AWS integrations, Genie workflows and

Databricks features such as Unity Catalog, Delta Lake optimizations, Workflow orchestration, and advanced cluster management. The ideal candidate has both data engineering and cloud engineering expertise.

Key Responsibilities

Databricks Platform Engineering

• Build, configure, and maintain Databricks Jobs, Workflows, Pipelines, and Delta Tables.
• Develop scalable high-performance ETL pipelines using PySpark, SQL, and Databricks notebooks.
• Apply optimization techniques - SQL queries, Delta Lake tables, partitioning strategies, liquid clustering.
• Ensure workflows scale efficiently and maintain reliability.
• Manage cluster performance, job debugging, resource auto-scaling.
• Document workflows, data models
• Build, publish and maintain Databricks Dashboards
• Apply the Databricks Backup and restore methods based on requirements.

Security, Governance & Compliance

• Implement Unity Catalog, data lineage, access controls, and governance best practices.
• Manage secrets, tokens, and secure integrations.
• Ensure compliance with enterprise security and data governance standards.

AWS & Databricks Integration

• Integrate Databricks with AWS services (S3, EC2, IAM).
• Manage roles, secure data lake permissions.
• Optimize cluster configuration for cost, performance, and scalability on AWS.

Skill Requirements

Technical Skills - Working with Genie, debugging skills, CI/CD experience and good understanding of networking and security practices.

Other Requirements

Nice-to-Have Skills

AI and Machine Learning capabilities in Databricks.