Role: Databricks Engineer

Experience

4–8 years

Location

As per business requirement

Employment Type

Full-time

Job Summary

We are looking for a skilled
Databricks Engineer
with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Key Responsibilities

• 
  Design, develop, and maintain scalable data pipelines using Databricks.
• 
  Build ETL/ELT workflows for batch and streaming data processing.
• 
  Develop solutions using PySpark, Spark SQL, and Delta Lake.
• 
  Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
• 
  Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
• 
  Optimize Spark jobs for performance, scalability, and cost efficiency.
• 
  Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
• 
  Implement CI/CD pipelines and deployment automation for Databricks workloads.
• 
  Ensure data quality, security, governance, and compliance.
• 
  Monitor, troubleshoot, and optimize production data pipelines.
• 
  Document technical solutions and follow engineering best practices.

Required Skills

Core Technologies

• 
  Databricks Lakehouse Platform
• 
  Apache Spark
• 
  PySpark
• 
  Spark SQL
• 
  Delta Lake
• 
  Python
• 
  SQL

Cloud Platforms (one or more)

• 
  Microsoft Azure (preferred)
• 
  AWS
• 
  Google Cloud Platform

Azure Technologies (Preferred)

• 
  Azure Data Factory (ADF)
• 
  Azure Data Lake Storage (ADLS Gen2)
• 
  Azure Synapse Analytics
• 
  Azure Key Vault
• 
  Azure DevOps

Data Engineering

• 
  Data Warehousing
• 
  Data Modeling
• 
  ETL/ELT Development
• 
  Batch Processing
• 
  Streaming (Kafka/Event Hubs)
• 
  Data Lake Architecture

DevOps & Version Control

• 
  Git
• 
  Azure DevOps / GitHub
• 
  CI/CD Pipelines

Preferred Qualifications

• 
  Experience with Unity Catalog.
• 
  Knowledge of Databricks Workflows and Jobs.
• 
  Hands-on experience with Delta Live Tables (DLT).
• 
  Exposure to MLflow is an added advantage.
• 
  Experience with data governance and security best practices.
• 
  Familiarity with Infrastructure as Code (Terraform) is a plus.

Educational Qualification

• 
  Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Certifications

• 
  Databricks Certified Data Engineer Associate
• 
  Databricks Certified Data Engineer Professional
• 
  Microsoft Certified: Azure Data Engineer Associate (DP-203)
• 
  Azure Fundamentals (AZ-900)

Good to Have

• 
  Experience with real-time analytics.
• 
  Knowledge of Lakehouse architecture.
• 
  Experience with Agile/Scrum methodologies.
• 
  Strong analytical and problem-solving skills.
• 
  Excellent communication and stakeholder management abilities.

Mandatory Skills

• 
  Databricks
• 
  PySpark
• 
  Spark SQL
• 
  Delta Lake
• 
  Python
• 
  SQL
• 
  Azure/AWS/GCP (at least one cloud platform)
• 
  ETL/ELT Development
• 
  Data Lake Architecture

Nice to Have

• 
  Unity Catalog
• 
  Delta Live Tables (DLT)
• 
  MLflow
• 
  Kafka/Event Hubs
• 
  Azure Data Factory
• 
  Terraform
• 
  Azure DevOps/GitHub Actions