Job Description

Job Summary

The Senior Technical Specialist will be responsible for creating, overseeing and managing ETL Pipelines, Azure Databricks, SQL, Pyspark. The primary objectives include optimizing data workflows, ensuring data reliability and integrity, and developing efficient data processing solutions using Azure technologies.

Key Responsibilities

1. Design, develop, and implement data pipelines using azure data factory (adf) or similar to extract, transform, and load data.
2. Utilize azure databricks to process and analyze large volumes of data efficiently.
3. Write complex sql queries to extract and manipulate data for reporting and analysis purposes.
4. Develop and maintain Pyspark scripts for data processing and automation tasks.
5. Collaborate with cross functional teams to troubleshoot data related issues and optimize data processes.
6. Stay updated on the latest azure technologies and recommend innovative solutions to improve data workflows.

Skill Requirements

1. Proficiency in building and scheduling ETL data pipelines.
2. Experience with azure databricks for processing big data workloads.
3. Strong skills in writing sql queries for data extraction and manipulation.
4. Proficient in Pyspark scripting for data processing and automation tasks.
5. Problem-solving skills with the ability to analyze complex data systems and processes.
6. Strong communication and teamwork abilities to collaborate effectively with different teams and stakeholders.

Other Requirements

1.Relevant certifications in Azure Data Factory (ADF), Azure Databricks, SQL, Pyspark are a plus.