Job Description

Job Summary

Design and implement RAG-based GenAI architectures on Databricks (ingestion, chunking, embeddings, vector search, retrieval). • Build and optimize LLM pipelines using Databricks (Workflows, MLflow, Unity Catalog, Model Serving). • Develop prompt engineering patterns, reusable prompt templates, and evaluation frameworks for LLM quality. • Integrate enterprise data sources into GenAI solutions while ensuring governance, lineage, and security via Unity Catalog. • Implement LLMOps practices: experiment tracking, model/version management, monitoring, and rollback strategies. • Collaborate with data engineers, MLOps engineers, and product teams to translate business problems into GenAI solutions. • Define and enforce coding standards, best practices, and design patterns for Databricks-based GenAI solutions. • Partner with security and compliance teams to ensure responsible AI use, data privacy, and adherence to internal policies. • Monitor and optimize cost, performance, and latency of LLM workloads on Databricks and AZURE cloud services.

Key Responsibilities

1. To be responsible for providing technical guidance to a team of developers, enhancing their technical capabilities and increasing productivity.
2. To conduct comprehensive code reviews, establish and oversee quality assurance processes, performance optimization , implementation of best practices and coding standards to ensure successful delivery of complex projects.
3. To ensure process compliance in the assigned module| and participate in technical discussions/review as a technical consultant for feasibility study (technical alternatives, best packages, supporting architecture best practices, technical risks, breakdown into components, estimations).
4. To collaborate with stakeholders to define project scope, objectives, deliverables and accordingly prepare and submit status reports for minimizing exposure & closure of escalations.

Skill Requirements

6–8+ years in data/ML engineering, with 2+ years on Databricks in production environments. • Strong hands-on experience with Databricks (PySpark, Databricks SQL, Workflows, MLflow, Model Serving, Unity Catalog). • Demonstrated experience building GenAI / LLM applications (RAG pipelines, chatbots, assistants, or content generation workflows). • Solid programming skills in Python (Databricks notebooks, modular code, unit tests). • Experience with vector databases / indexes (e.g., Databricks vector search, or equivalent) and embeddings. • Knowledge of LLM providers (e.g., Azure OpenAI, OpenAI, other major cloud LLMs) and integration patterns. • Strong understanding of MLOps / LLMOps concepts: CI/CD, monitoring, observability, model lifecycle management. • Excellent communication skills, with ability to explain complex AI concepts to non technical audiences.

Other Requirements

Experience with GenAI patterns: RAG, tool/function calling, agents/orchestrators. • Background in feature engineering, classical ML, and experimentation frameworks. • Experience with governed AI in regulated environments • Familiarity with Delta Lake, Databricks SQL, and BI integration. • Knowledge of policy and safety tooling (guardrails, content filters, red-teaming approaches). Example Role Outcomes (6–8 Months) • Delivered 1–2 production-grade GenAI solutions on Databricks with measurable business value. • Established reference architectures and best practices for Databricks-based GenAI across teams. • Reduced time-to-market for new GenAI use cases by building reusable pipelines, templates, and tooling.