Job Description

Job Summary

We are seeking an experienced Databricks Administrator with 7+ years of overall IT experience and strong expertise in managing, securing, monitoring, and optimizing Databricks environments on Azure, AWS, or GCP. The ideal candidate should have hands-on experience in platform administration, cluster management, security governance, monitoring, automation, and integration with cloud services.

Key Responsibilities

Key Responsibilities

• Administer and manage Databricks workspaces across development, testing, and production environments.
• Configure and maintain Databricks clusters, cluster policies, instance pools, and job scheduling.
• Manage user access, permissions, RBAC, Unity Catalog, and security policies.
• Implement governance, compliance, and data security best practices.
• Monitor platform performance, resource utilization, and cost optimization.
• Troubleshoot platform, connectivity, performance, and job execution issues.
• Integrate Databricks with cloud-native services such as Azure AD, AWS IAM, Key Vault, Storage Accounts, ADLS, S3, etc.
• Manage networking components including VNet Injection, Private Link, NSGs, Firewalls, and secure connectivity.
• Automate administrative tasks using Terraform, PowerShell, Python, CLI, or REST APIs.
• Establish monitoring and alerting using Azure Monitor, Log Analytics, CloudWatch, or equivalent tools.
• Support CI/CD implementation and infrastructure provisioning.
• Collaborate with Data Engineering, Data Science, Security, and Infrastructure teams.
• Ensure platform availability, backup, disaster recovery, and operational excellence.

Skill Requirements

Required Skills

Databricks Administration

• Databricks Workspace Administration
• Cluster Configuration & Management
• Unity Catalog Administration
• Job Scheduling & Workflow Management
• Access Control & Security Governance
• Cost Optimization & Monitoring

Cloud Platforms (Any One Preferred)

• Microsoft Azure Databricks
• AWS Databricks
• Google Cloud Databricks

Cloud Services

• Azure AD / Entra ID
• ADLS Gen2 / Blob Storage
• Azure Key Vault
• AWS IAM, S3, Secrets Manager
• Networking & Security Services

Automation & DevOps

• Terraform
• Azure DevOps / GitHub Actions
• Databricks CLI
• REST APIs
• PowerShell / Shell Scripting
• Python

Monitoring & Security

• Azure Monitor / Log Analytics
• CloudWatch
• SIEM Integration
• Audit Logs & Compliance Controls

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field.
• 7+ years of overall IT experience.
• 3+ years of hands-on Databricks Administration experience.
• Strong experience in cloud platform administration (Azure preferred).
• Experience with IaC tools such as Terraform.
• Strong understanding of networking, security, and identity management.

Other Requirements