Job Description

Job Summary

Plutus21 is seeking an AI Engineer to join our core platform team and take ownership of agent evaluation and governance. The AI Engineer will build the systems that test, approve, and monitor AI agents before and after they go live, and will ensure that governance rules are enforced in code at runtime. This role works closely with the Principal Platform Engineer, the Cloud Engineer, and product and client engagement teams, and is critical to preparing AI capabilities for use by external clients.

Key Responsibilities

• Build and maintain an evaluation framework covering datasets, metrics, agreed thresholds, and a go-live gate for each agent capability.
• Run re-evaluations whenever an agent's model or instructions change, and identify regressions before release.
• Build a governed agent runtime that assigns an accountable owner to each action, requires human approval where needed, and applies deterministic checks before any irreversible action.
• Implement rule-level and campaign-level approval for high-volume actions, including statistically sized review samples before release and automatic suspension when quality thresholds are breached.
• Design review and evaluation workflows for AI-generated content that is sent outside the client organization.
• Ensure every model call is recorded with the model used, instruction version, data provenance, and quality checks applied.
• Keep the platform independent of any single model provider.
• Work with the Principal Platform Engineer to convert written governance rules into executable validation.
• Collaborate with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
• Document design decisions clearly before implementation begins.

Qualifications

• Proven experience shipping tool-using, multi-step LLM agents to production environments, beyond prototypes.
• Demonstrated experience building evaluation sets and metrics, setting thresholds, and catching regressions across model or prompt changes.
• Working knowledge of applied statistics, including sample sizing, confidence bounds, and estimating error rates from samples.
• Experience designing systems in which rules such as approvals, checks, and kill switches are enforced in code rather than left to the model.
• Strong software engineering skills in Python or TypeScript, including testing, code review, and building maintainable services.
• Strong written communication skills, with the ability to produce clear design notes and work from documented decisions.
• Experience with knowledge graphs or ontologies (such as LinkML, SHACL, or OWL/RDF) and translating domain rules into validation is a plus.
• Experience building human-in-the-loop systems, including review queues, escalation paths, and approval workflows, is a plus.
• Experience with retrieval over governed data, including access control and data classification, is a plus.
• Experience in regulated domains such as financial services or privacy-sensitive environments is a plus.
• Experience working with more than one model provider is a plus.