Blue Pearl HQ

Data/AI Architect (Databricks)

Blue Pearl HQ  •  Johannesburg, ZA (Onsite)  •  6 hours ago
Apply
AI can make mistakes so check important info. Chat history is never stored.

Job Description


The key focus for the
senior data/AI architect is to perform planning aligned to key AI solutions,
build and participate in the architecture capability building, perform AI
architecture and design, manage AI architecture risk and compliance, provide
design and build governance and support and communicate and share knowledge
around the architecture practices, guardrails, blueprints and standards related
to the AI solution design.


A key focus of this role is
partnering with the AI Technology Centre of Excellence to build out the
organisation's Databricks AI platform and support the delivery of enterprise AI
and generative AI use cases.


Planning


  • Lead AI solution
    requirements gathering and ensure alignment with business objectives and
    constraints.

  • Define and refine AI
    architecture runways for intentional architecture with the key
    stakeholders

  • Provide input into
    business cases and costing

  • Participate and
    provide AI architectural runway requirements into Programme Increment (PI)
    Planning


Architecture Capability


  • Design and implement enterprise-grade AI
    architectures leveraging Databricks and cloud-native technologies.

  • Develop and
    oversee AI architecture views and ensure alignment with enterprise
    architecture.

  • Maintain and oversee
    the AI solution artifacts in the set enterprise repository and knowledge
    portals aligned to the rest of the architecture

  • Manage the AI
    architecture processes based on the requirements for each architype

  • Manage change impact
    of the AI architecture with stakeholders

  • Develop and
    participate in the build of the AI architecture practice with embedded
    architects and engineers including the relevant methods, repository and
    tools

  • Manage the AI
    architecture considering the business, application, information/data and
    technology viewpoints

  • Establish, enforce and
    implement AI standards, guardrails, frameworks, and patterns

  • Partner with the AI
    Tech COE to define and evolve the Databricks AI platform architecture,
    ensuring alignment with enterprise data and AI strategy

  • Design and implement
    AI/ML architectures on Databricks, including MLOps pipelines, model
    lifecycle management, Unity AI Gateway and Unity Catalog governance for
    AI/ML assets


Solution Design


  • Lead and
    review logical and detailed AI architecture

  • Evaluate and
    approve AI solution options and technology selections

  • Select appropriate
    technology, tools and build for the solution

  • Oversee and
    maintain the AI solution blueprints

  • Drive incremental
    modernisation initiatives in the delivery area

  • Design and evaluate
    architectures for AI and generative AI use cases, including RAG pipelines,
    vector stores, feature stores, and LLM integration patterns


Risk, Governance and Compliance


·

Identify, assess and mitigate risks at a AI solution architecture
level


·

Ensure and enforce compliance with policies, standards, and
regulations


·

Lead AI architecture reviews and integrate with governance
functions


·

Integrate with other governance and compliance functions to ensure
continuity in managing the investment and risk for the organisation pertaining
to the solution architectures


·

Establish and provide AI standards, guidance, and tools to delivery
teams.


Implementation and Collaboration


·

Establish and provide AI solution architectures and tools to the
delivery and AI engineering teams


·

Lead and facilitate collaboration with delivery teams to achieve
architecture objectives


·

Manage and resolve deviations and ensure up-to-date AI solution
design documentation


·

Identify opportunities to optimise delivery of solutions


·

Oversee and conduct post-implementation reviews


·

Ensure the AI architecture supports CI/CD pipelines to facilitate rapid
and reliable deployment of data solutions


·

Implement automated testing frameworks for AI solutions to ensure
quality and reliability throughout the development lifecycle.


·

Establish performance monitoring and optimisation practices to ensure AI
solutions meet performance benchmarks and can scale as needed.


·

Integrate robust AI security measures, including encryption, access
controls, and regular security audits, into the implementation process.


Communication and Knowledge Sharing


·

Communicate and advocate up-to-date AI solution architecture views


·

Communicate the relevant AI standards, practices, guardrails and tools
to stakeholders relevant to the solution design


·

Ensure IT teams are well-informed and trained in architecture
requirements


·

Communicate and collaborate with stakeholders' relevant views on
planning, technology assessments, risk, compliance, governance and
implementation assessments


·

Foster collaboration between AI architects, AI
engineers, and other IT teams through regular cross-functional meetings and
agile ceremonies.


·

Communicate and maintain up-to-date blueprint designs for key data
solutions


·

Ensure effective participation in the agile ceremonies (PI planning,
sprint planning, retrospectives, demos)


·

Implement regular feedback loops with stakeholders
and end-users to continuously improve data solutions based on real-world usage
and requirements


·

Create a culture of knowledge sharing by organising
regular workshops, training sessions, and documentation updates to keep all
team members informed about the latest AI architecture practices and tools


Requirements


MINIMUM
QUALIFICATIONS/EXPERIENCE


  • Matric

  • Degree or diploma in
    Information Technology, Computer Science, Engineering OR relevant diploma
    / degree

  • Experience:

    Requires a minimum of
    5 years in a technical/solution design role and a minimum of 7 years
    relevant IT experience

  • Data and AI
    Experience:
    Required a minimum of 7 years related experience in AI,
    data engineering, data modeling and design and data management and
    governance

  • Expert-level proficiency in Databricks, including
    Delta Lake, Spark, and MLflow.


·

Proven
experience architecting and delivering AI/ML solutions on Databricks, including
MLOps, model deployment and monitoring, and Unity Catalog governance for AI/ML
assets.


·

Hands-on
experience in large-scale data and AI platform implementation (preferably
cloud-based).


ADDITIONAL
QUALIFICATIONS/EXPERIENCE (PREFERRED, NOT A REQUIREMENT)


  • DAMA-DMBOK

  • TOGAF

  • ArchiMate

  • Cloud Certifications
    (AWS, Azure)

  • Financial Industry
    Experience

  • Certifications in Databricks, AWS ML, AWS Data
    Engineering or similar.

  • Experience with
    generative AI / LLM architectures (e.g. RAG pipelines, vector databases,
    AI gateways)

  • Databricks Certified
    Machine Learning Professional or equivalent certification


Data
Related Experience:


·

Big Data and Analytics (e.g., Hadoop, Spark)


·

Data Warehousing


·

Master Data Management (MDM)


·

Data Lakes, Lakehouse, and Data Mesh


·

Metadata Management


·

ETL/ELT Processes


·

Data Privacy and Compliance


  • Cloud Data Services

  • Experience with AI cloud platforms (Azure, AWS,
    or GCP) and associated data services.


·

Proficiency in
SQL, Python, and distributed data processing frameworks.


·

Familiarity with
CI/CD for data pipelines and DevOps practices.


·

Experience with
Lakehouse architecture and real-time streaming solutions


Related
attributes and competencies related to architecture:


·

Critical thinking/problem solving


·

Teamwork/collaboration


·

Effective Communication Skills


·

Leadership skills


·

Knowledge and experience in architecture domains


·

Knowledge and experience in architecture methods, frameworks and tools


·

Solution Design Experience


·

Agile Knowledge and Experience


·

Cloud Knowledge and Experience


AI
related competencies:


·

AI architecture principles and methodologies


·

AI integration technologies and tools


·

AI management and governance


·

AI/ML architecture, MLOps, and model lifecycle management knowledge and
experience

Blue Pearl HQ

About Blue Pearl HQ

Blue Pearl is a market-leading CLOUD Solutions developer with extensive knowledge and insight into the latest technologies, standardised processes, advanced technical capabilities and consulting processes available, ensuring wholistic success for our clientele. We offer professional consulting to compliment your business strategy and overall management and make it our priority to add value to any business by listening, analysing and creating a conducive solution that will empower our client.

We implement a Data Analysis Process that includes inspecting, cleansing, transforming, and modelling data with the end-goal of discovering useful information, informing conclusions, and relevant information to support your decision-making. Your business cannot afford not to engage with us, allowing our data analysis to play a role in making your business decisions more scientific and helping your business achieve effective operation.

Blue Pearl’s team of experts include BI strategists, BI analysts, Data Warehouse Architects, Data Scientists, Implementation and Development experts. With the use of BI, Analytics and Big Data, we effectively partner with our customers on their mission to achieve a competitive business advantage and real ROI from the structured information we collect.

Industry
IT & Software
Company Size
11-50 employees
Headquarters
Johannesburg, ZA
Year Founded
2013
Social Media