Job Description

Job Summary

We are seeking a seasoned Databricks Tech Lead to design, build, and scale our next-generation data ingestion and processing framework. In this role, you will lead a team of data engineers, champion best practices in software development, and build reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into our Lakehouse platform.

Key Responsibilities

Key Responsibilities

• Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
• Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
• Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
• Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
• CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
• Governance & Security: Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Skill Requirements

1. Proficiency in azure data factory (adf) and azure databricks for building and managing data pipelines.
2. Strong experience with sql and oracle pl/sql for data querying and manipulation.
3. Advanced programming skills in python for scripting and data processing tasks.
4. Knowledge of data modeling, data warehousing concepts, and database design principles.
5. Ability to work in a collaborative team environment and communicate effectively with stakeholders.
6. Strong analytical and problem-solving skills with attention to detail.
7. Experience in data visualization tools and techniques is a plus.

Other Requirements

1.Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle PL/SQL, or Python are advantageous.