Job Description

We are seeking a highly skilled Senior Data Engineer to design, develop, and maintain robust data pipelines that extract, transform, and load data from disparate applications into our Azure-based data platform. The ideal candidate will have a strong foundation in data engineering principles, a deep understanding of cloud technologies, and a passion for solving complex data challenges.

Roles and Responsibilities

Design and develop scalable and efficient data pipelines using Azure Data Factory, Azure Databricks, Kafka, and other relevant tools.

Extract data from various applications using APIs, webhooks, or other methods.

Ingest and process real-time data streams using Kafka.

Transform and clean data to ensure data quality and consistency.

Load data into ADLS Gen2 in optimized formats (e.g., Parquet, Delta Lake).

Build and maintain data models and schemas.

Collaborate with data analysts and business stakeholders to understand data requirements and deliver actionable insights.

Optimize data pipelines for performance and cost-efficiency.

Implement data governance and security best practices.

Mentor junior data engineers and provide technical guidance.

Skills and Qualifications

Strong proficiency in Python, Scala, or Java programming languages.

Expertise in data engineering tools and technologies such as Azure Data Factory, Azure Databricks, ADLS Gen2, Kafka, and Delta Lake.

Experience with cloud computing platforms, preferably Azure.

Knowledge of data modelling, ETL/ELT processes, data warehousing, and real-time data processing concepts.

Strong problem-solving and analytical skills.

Ability to work independently and as part of a team.

Excellent communication and interpersonal skills.

Bachelor's degree in computer science, engineering, or a related field.

Preferred Qualifications

Knowledge of machine learning and AI concepts.

Certifications in Azure or related technologies.

Experience with Node.js, Java or .NET Core for developing connector APIs.

Must-Have Technologies

Python

Scala or Java

Azure Data Factory

Azure Databricks

ADLS Gen2

Kafka

Delta Lake

SQL

This role offers a unique opportunity to work on challenging data engineering projects and contribute to the growth of our data-driven organization.