Job Description

The Senior Data Engineer is responsible for designing, building, and managing scalable data pipelines and data platform solutions within the Enterprise Data Platform (EDP) using Azure and Databricks. The role focuses on enabling reliable, governed, and high-performance data integration and transformation across multiple source systems to support analytics, reporting, and advanced data use cases.

Duties and Responsibilities

This position combines hands-on data engineering, platform optimization, and stakeholder coordination, ensuring the delivery of robust, secure, and business-aligned data solutions while supporting data governance and best practices.

Data
Engineering & Pipeline Development

• Design, develop, and maintain
scalable data pipelines using Azure Databricks, Azure Data Factory, and related
services

• Implement ETL/ELT processes to
ingest, transform, and load data into the Enterprise Data Platform

• Develop batch and near real-time
data processing solutions

• Apply medallion architecture
(bronze, silver, gold layers) for structured data processing

• Ensure data quality, consistency,
and integrity across pipelines and datasets

Platform
Operations & Optimization

• Monitor, maintain, and optimize
data pipelines, workflows, and jobs in Databricks and Azure

• Troubleshoot pipeline failures,
performance issues, and data inconsistencies

• Optimize Spark jobs, query
performance, and storage utilization (e.g., Delta Lake optimization)

• Ensure high availability,
scalability, and reliability of data services

• Maintain documentation and ensure
compliance with data governance, security, and operational standards

Team
Management

• Provide guidance to data engineers
and developers on best practices in Azure and Databricks

• Review code, pipelines, and
technical designs to ensure quality and standards compliance

• Support the team in resolving
technical issues and improving delivery efficiency

• Promote reusable frameworks,
templates, and automation practices

• Mentor team members on data
engineering concepts and cloud-based architectures

Stakeholder
Engagement

• Collaborate with business units,
analytics teams, and platform engineers to gather and validate data
requirements

• Translate business needs into
scalable data models and pipelines

• Act as liaison between source
system owners, data consumers, and platform teams

• Communicate pipeline status,
issues, and improvements to stakeholders

• Support analytics and BI teams by
ensuring availability of clean and reliable datasets

Technical Competencies and Skills

Data Engineering & Databricks, Azure Data Platform, Data Modeling & Performance Optimization, Data Governance & Security, DevOps & Automation

Education, Trainings and Licenses Required

• Bachelor's degree in Computer Science, Information Technology