Job Description

Job Title: Technical Lead / Solution Architect – Data & Analytics
About the Role
We are seeking a Technical Lead/Solution Architect to own the architecture and technical delivery of our enterprise Data & Analytics platform. You will be the bridge between business strategy and technical execution.
In this role, you will define the architectural roadmap for our cloud-native data ecosystem and lead a team of Data Engineers and Data Scientists through the implementation lifecycle. You will leverage the full power of the Databricks Lakehouse Platform and Azure to build scalable, high-performance data pipelines and analytical solutions that drive critical business decisions.
Key Responsibilities
1. Architecture & Strategy

• Design & Blueprint: Define the end-to-end architecture for data ingestion, storage, processing, and serving using Azure (Data Lake Gen2, Synapse) and Databricks (Delta Lake, Unity Catalog).
• Technology Selection: Evaluate and select appropriate Azure services and open-source tools to solve complex business problems, ensuring cost optimization and performance scalability.
• Data Governance: Implement robust data governance, security, and compliance frameworks using Unity Catalog and Azure Purview.
• Migration Strategy: Lead the architectural design for migrating legacy on-premise workloads to the modern Azure/Databricks stack.

2. Technical Leadership & Implementation

• Hands-On Guidance: Act as a player-coach—providing hands-on technical leadership to the implementation team during the build phase (coding complex pipelines, optimizing Spark jobs).
• Code Quality: Define and enforce coding standards, CI/CD pipelines (Azure DevOps/GitHub Actions), and automated testing strategies for data applications.
• Performance Tuning: Optimize Apache Spark jobs, Delta Lake performance, and cluster configurations for peak efficiency and cost-effectiveness.
• Mentorship: Mentor mid-level and junior engineers in software engineering best practices, distributed computing patterns, and cloud-native development.

3. Collaboration & Stakeholder Management

• Cross-Functional Alignment: Collaborate with Data Scientists, Product Managers, and Business Analysts to translate functional requirements into technical specifications.
• DevOps Integration: Work closely with DevOps teams to ensure infrastructure is reliable, secure, and monitored using Azure Monitor and Databricks monitoring tools.
• Documentation: Create and maintain high-level architecture diagrams, data flow diagrams, and technical runbooks.

Required Qualifications
Experience

• 6+ years of experience in Data Engineering, or Solution Architecture.
• 3+ years of experience specifically designing and implementing solutions.
• Proven experience architecting and building large-scale data platforms utilizing Databricks(Lakehouse architecture).
• Deep expertise in Apache Spark (PySpark or Scala) for big data processing.

Technical Skills

• Cloud: Azure Data Factory, Azure Data Lake Storage (ADLS) Gen2, Azure Synapse Analytics, Azure Key Vault.
• Data Engineering: Advanced SQL, Python (Pandas, PySpark), Delta Lake, Parquet.
• DevOps: Infrastructure as Code (Terraform/Bicep), CI/CD pipelines, and containerization (Docker/Kubernetes).
• Architecture Patterns: Lambda/Kappa architectures, Medallion Architecture (Bronze/Silver/Gold), Event-driven architectures (Event Hubs/IoT Hub).

Leadership Skills

• Proven ability to lead technical teams of 5-10 engineers in an Agile/Scrum environment.
• Excellent communication skills with the ability to simplify complex technical concepts for non-technical stakeholders.
• Experience with technical estimation and resource planning.

Nice to Have

• Databricks Certification (e.g., Databricks Certified Architect Professional).
• Microsoft Azure Solutions Architect Expert (AZ-305).
• Experience with Real-time streaming technologies (Kafka, Azure Stream Analytics).
• Knowledge of Machine Learning lifecycle (MLflow) and MLOps practices.