Job Description

Data Integration Lead Responsibility Design and implement scalable data pipelines for ingesting and transforming data primarily using Databricks and leveraging PySpark notebooks, Spark SQL, and Python. Develop ETL processes to extract, transform, and load data from diverse sources into data lakehouse architecture on Databricks Analyze existing integration Terdata landscape (TPT, BTEQ, Talend, IBM Sterling) Define ingestion and integration strategy for Databricks Ensure seamless data flow from source systems to Lakehouse Lead integration design and oversee pipeline migration Optimize data processing workflows to enhance performance and efficiency using Databricks capabilities. Ensure data security and compliance with data privacy regulations throughout the data engineering process. Deliver high quality data products based on business requirements Collaborate with stakeholders to gather, understand requirements and create technical solutions using Microsoft Azure stack Create comprehensive documentation of workflows, pipelines, and architecture.