Job Description

Authority Partners is seeking an experienced Data Engineer to join our team. You will design, expand, and optimize data pipelines and architectures to support cross-functional teams. The ideal candidate is skilled in building and enhancing data systems, supporting developers, analysts, and scientists, and ensuring seamless data delivery across projects.

Up for the challenge? Then read on and apply!

Responsibilities:

• Design, build, and maintain reliable and scalable data pipelines for ingesting, transforming, and delivering structured and unstructured data.
• Implement batch and streaming data workflows that support real-time and analytical use cases.
• Optimize data flows for performance, cost-efficiency, and maintainability.
• Develop and maintain data models that support business intelligence, analytics, and operational needs.
• Ensure effective partitioning, indexing, and data lifecycle management strategies.
• Collaborate with analysts and business stakeholders to understand data requirements and deliver high-quality datasets and data products.
• Implement data quality checks, monitoring, and validation processes to ensure data accuracy and integrity.
• Ensure compliance with relevant data governance, security, and privacy standards.
• Monitor and troubleshoot data pipelines and systems to ensure reliability and high availability.
• Design and develop interactive dashboards and reports that provide actionable insights to stakeholders, ensuring data is presented in a clear, intuitive, and visually compelling manner.

What you have:

• Strong experience with Databricks for building scalable data pipelines and transformation workflows.
• Hands-on experience with PySpark for distributed data processing.
• Strong proficiency in SparkSQL for data transformation and analysis.
• Solid understanding of data pipelines and data modelling.

You stand out if you have:

• Experience working with Azure cloud services.
• Working knowledge of Databricks Dashboards.
• Exposure to data visualization tools (Power BI, Tableau, Looker).
• Experience with CI/CD pipelines for data engineering workflows.
• Working knowledge of Microsoft Fabric and Dynamics 365 Finance and Operations.

Education:

• Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field

About Authority Partners:

Authority Partners is a leading and global IT services company with over 25 years of experience, serving clients within the financial, healthcare, insurance, mortgage, and technology sectors among many others. We are committed to building relationships with IT professionals who possess integrity, dedication, teamwork, and the desire to learn and grow with us. We specialize in placement for long-term contracts and permanent opportunities and helping our consultants achieve their career success.

Authority Partners is an equal opportunity employer and is committed to providing a workplace that is free from all types of discrimination, as well as from abusive or offensive behavior and harassment.