An experienced data engineer with strong Azure Databricks expertise is required for a 6 month contract to develop and deliver a scalable data pipeline that ingests files and transforms them into standardised, high-quality datasets for reporting. This is a hands-on delivery role focused on building a production-grade pipeline, suited to someone comfortable working independently, defining technical solutions, and rapidly delivering high-quality data engineering components.
Location:
London - UK - Fenchurch
What will your job entail?
Key Responsibilities
• Implement a scalable ETL pipeline in Azure Databricks, that transforms raw data into standardised schemas to support reporting and analytics, following a medallion architecture in line with internal design standards.
• Leverage an existing email ingestion tool to extract multiple format source files from a shared mailbox.
• Develop notebooks, jobs and pipelines in Databricks for ETL orchestration.
• Design and implement robust validation rules, monitor pipeline runs and troubleshoot failures, with logging and notifications to support robust data processes.
• Integrate with CI/CD flows across environments using Azure DevOps.
• Produce clear technical documentation of the solution to ensure maintainability post-delivery.
Experience
• 2+ years hands-on experience with Azure Databricks, including data validation, transformation, and optimization.
• Expertise in PySpark/Spark SQL.
• Knowledge of Unity Catalog and modern Azure Databricks features, as well as data best practices and concepts.
• Proficiency building end-to-end data pipelines with Delta Lake/Lakehouse architecture.
• Experience implementing CI/CD pipelines (Azure DevOps preferred).
• Experience supporting BI reporting layers (Power BI or similar) is nice to have.
• Insurance experience is nice to have.
Disclaimer
Ryan Specialty is an Equal Opportunity Employer. We are committed to building and sustaining a diverse workforce throughout the organization. Our vision is an inclusive and equitable workplace where all employees are valued for and evaluated on their performance and contributions. Differences in race, creed, color, religious beliefs, physical or mental capabilities, gender identity or expression, sexual orientation, and many other characteristics bring together varied perspectives and add value to the service we provide our clients, trading partners, and communities. This policy extends to all aspects of our employment practices, including but not limited to, recruiting, hiring, discipline, firing, promoting, transferring, compensation, benefits, training, leaves of absence, and other terms, conditions, and benefits of employment.
Ryan Specialty (NYSE: RYAN) is an international specialty insurance firm that provides distribution, underwriting, product development, administration and risk management services by acting as a wholesale broker and a managing underwriter. Our mission is to provide industry-leading innovative specialty insurance solutions for insurance brokers, agents and carriers.
Ryan Specialty was founded in 2010 by Patrick G. Ryan, the widely respected insurance veteran, founder and former Chairman and CEO of Aon Corporation. Mr. Ryan saw the need in the marketplace for a specialty organization dedicated exclusively to creating sophisticated insurance solutions for risks that demand innovation, proven leadership, outstanding industry expertise and quality services on behalf of brokers, agents and carriers.
For more information about Ryan Specialty, visit our website at ryansg.com.
Ryan Specialty is an Equal Opportunity Employer. We are committed to building and sustaining a diverse workforce that is reflective of society throughout the entirety of the organization. Our vision is an inclusive and equitable workplace where all employees are valued and evaluated on their performance and contributions. Differences in race, creed, color, religious beliefs, physical or mental capabilities, gender identity or expression, sexual orientation, and many other characteristics bring together varied perspectives and add value to the service we provide our clients, trading partners, and communities. This policy extends to all aspects of our employment practices, including but not limited to, recruiting, hiring, discipline, firing, promoting, transferring, compensation, benefits, training, leaves of absence, and other terms, conditions and benefits of employment.
