Job Description

Principal Data Engineer - Principal Consultant

Location: Queensland (Brisbane preferred) — OR — Anywhere in Australia (Sydney preferred)

About Us

At Agile Insights, we are passionate about empowering organisations to transform through the power of Data & AI. We work with global brands and leading organisations to deliver innovative solutions that solve real business challenges. We have a fantastic, long-standing relationship with Microsoft and are extremely proud to be a Microsoft Solutions Specialist across four competencies, including Data Analytics and AI, Azure, and Digital & App Innovation.

Your Role

We are seeking an experienced and dynamic Principal Data Engineer to lead data engineering engagements, design scalable modern data solutions, and help clients unlock the full potential of their data. This is a senior, client-facing role that balances deep technical expertise with leadership, advisory, and relationship-building responsibilities.

You will act as a trusted advisor to our clients — shaping data strategy, leading architecture discussions, guiding delivery teams, and representing Agile Insights in the market. We're looking for someone who is genuinely hands-on and stays current with the data engineering landscape (including how AI can be used to augment engineering work), but who equally thrives in the non-technical aspects of consulting: client relationships, communication, leadership, and advisory.

Key Responsibilities

Technical Leadership & Architecture

• Lead solution architecture for modern data platforms, with a focus on Microsoft Fabric and the broader Azure Data ecosystem.
• Design and oversee the implementation of scalable, secure, and high-performing data pipelines, lakehouses, data warehouses, and integration solutions.
• Advise clients on best practices, governance frameworks, and modern data architecture patterns (medallion architecture, lakehouse, data mesh, etc.).
• Drive innovation by leveraging Microsoft services such as Microsoft Fabric, Azure Synapse, Azure Data Factory, Azure Data Lake, Azure SQL, and Microsoft Purview for governance.
• Stay hands-on: review designs and code, prototype solutions, and unblock teams when needed.
• Champion the responsible use of AI tools to augment engineering work — improving productivity, code quality, and delivery velocity across teams.

Client Engagement & Advisory

• Act as a trusted advisor to clients, providing insights and recommendations on data strategy, architecture, and modernisation roadmaps.
• Build and nurture long-term client relationships, becoming the go-to person clients call when they need senior data engineering guidance.
• Conduct workshops, technical demonstrations, assessments, and proof-of-concept (POC) projects to showcase the value of modern data platforms.
• Translate complex technical concepts into clear business outcomes for executive and non-technical stakeholders.
• Identify opportunities to expand engagements and contribute to the growth of existing accounts.

Project Technical Leadership & Delivery

• Provide technical leadership across multiple concurrent engagements, ensuring quality, scope, and timelines are met.
• Mentor and guide project teams, supporting task delegation, technical decision-making, and team growth.
• Collaborate with delivery managers and stakeholders to shape technical requirements and align solutions with business goals.
• Set the bar for engineering excellence — code quality, testing, CI/CD, documentation, and operational readiness.

Innovation, Community & Knowledge Sharing

• Stay current with emerging trends across the Microsoft Data & AI stack (Fabric, Databricks, Purview, Copilot/AI tooling) and share that knowledge internally.
• Develop reusable frameworks, accelerators, and best practices to enhance delivery efficiency across the practice.
• Contribute to thought leadership through whitepapers, blogs, conference talks, and community events.
• Collaborate with Microsoft and other partners to co-develop solutions and secure funding for client initiatives.

Practice Growth

• Support pre-sales efforts by crafting proposals, developing demos, and estimating delivery costs.
• Represent Agile Insights at industry events, panels, and conferences.
• Build and nurture a network of client and partner relationships, identifying opportunities to expand our presence in the market.

What We're Looking For

This role is as much about how you work with people as it is about what you know technically. The ideal candidate combines deep hands-on data engineering skills with the maturity, communication, and leadership presence expected of a Principal Consultant.

Experience

• 8+ years of experience in data engineering, including significant time delivering data solutions on Microsoft Azure.
• Proven track record delivering large-scale, enterprise-grade data solutions in a consulting or client-facing environment.
• Experience leading project teams, owning architecture decisions, and acting as the senior technical voice in client engagements.

Technical Expertise

• Microsoft Fabric — strong hands-on experience designing and delivering Fabric-based solutions (Lakehouse, Warehouse, Data Factory, Real-Time Intelligence, OneLake).
• Azure Data Solutions — solid experience across Azure Data Factory, Azure Data Lake, Azure Synapse, Azure SQL, and related services.
• Strong proficiency in data modelling, ETL/ELT design, and building scalable, well-governed data pipelines.
• Proficiency in Python, PySpark, and SQL
• Hands-on experience with CI/CD for data platforms (Azure DevOps, GitHub Actions, or similar).
• Working knowledge of cloud security, compliance, and data governance frameworks.
• Practical experience using AI tools (e.g. GitHub Copilot, Claude, ChatGPT, Copilot in Fabric) to accelerate engineering work — and a point of view on how to apply them well in client delivery.

Good to Have

• Databricks — hands-on experience with Databricks on Azure (Unity Catalog, Delta, workflows).
• Microsoft Purview — experience implementing data governance, cataloguing, and lineage.
• Familiarity with Power BI and the broader Microsoft analytics ecosystem.
• Exposure to data mesh, data product thinking, or federated governance models.

Certifications (preferred)

• Microsoft Certified: Fabric Data Engineer Associate / Fabric Analytics Engineer Associate
• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert
• Databricks certifications (Data Engineer Associate/Professional) — a plus
• Other relevant Azure / Data & AI certifications

Non-Technical Skills (just as important as the technical)

• Client relationship management — you build trust quickly and maintain long-term relationships with senior stakeholders.
• Communication & presentation — you can comfortably present to executives, run workshops, facilitate architecture discussions, and write clearly.
• Leadership — you mentor engineers, lift teams around you, and set the technical and cultural bar.
• Technical project leadership — you can own the technical success of an engagement end-to-end, balancing scope, quality, and delivery.
• Architecture & advisory mindset — you think beyond the immediate ticket and help clients make good long-term decisions.
• Problem-solving and pragmatism — you balance ideal solutions with what's right for the client's context, budget, and capability.

What's In It for You?

At Agile Insights, we believe in nurturing talent and creating a supportive, flexible, and rewarding workplace. Here's what you can look forward to:

• Wellbeing: Hybrid work arrangements, an Employee Assistance Program, and an individualised gym and wellbeing program.
• Flexibility: Work from anywhere for up to 6 weeks per year.
• Professional Development: Additional paid leave for certifications and exams, with Agile-sponsored training programs for Microsoft and Databricks certifications.
• Team Engagement: Regular team meetups, including state-based gatherings and an annual all-company event.
• Equipment & Hardware: An allowance for your home office setup and a 2-year equipment incentive.
• Salary Packaging: Access to novated leasing services for tax-effective car purchases.
• Learning Resources: Access to our digital library and learning platforms, plus one-on-one mentoring with senior consultants.

Why Agile Insights?

• Work with some of the largest clients and global brands across diverse industries.
• Be part of a fast-growing, award-winning team passionate about transforming organisations with Data & AI.
• Thrive in a friendly start-up culture that values innovation, collaboration, and work-life balance.

Ready to Lead with Us?

If you're passionate about data engineering, consulting, and helping clients succeed, we'd love to hear from you.

Apply now to join Agile Insights and take your career to the next level!

No Agencies — No overseas applications — Must be in Australia and have full work rights.