AI Evaluation & Test Design

• Translate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows, and edge cases

• Design adversarial and boundary test inputs to expose hallucination, misclassification, and failure modes

• Validate AI outputs for structure, consistency, accuracy, and production readiness against defined performance thresholds

Evaluation Engineering & Automation

• Build reusable Python-based evaluation frameworks, including output validation, hallucination detection, and scoring mechanisms

• Develop parameterized test scripts reusable across features, models, and releases

• Implement AI-as-Judge frameworks, including prompt design, scoring logic, and calibration of evaluation reliability

• Embed evaluation frameworks into CI/CD pipelines to support continuous testing and deployment

Drift Detection & Quality Monitoring

• Design andoperatedrift detection frameworks using fixed baseline datasets and scheduled re-evaluation

• Establish thresholds to distinguish acceptable variation from performance degradation

• Enable release gating byidentifyingregressions prior to production deployment

Ground Truth & Data Quality

• Build andmaintainground truth datasets in partnership with subject matter experts

• Define standards for classification, extraction accuracy, and acceptable output characteristics

• Continuously update datasets to reflect evolving business requirements and use cases

Workflow & Integration Testing

• Test end-to-end agentic workflows,validatingdata integrity, error propagation, and fallback behavior

• Perform API-level testing of AI pipeline endpoints using Python and Postman/Newman

• Validate data persistence and integrity across system layers using SQL

• Partner with engineering teams to ensure testability, observability, and system reliability

Standardization & Scaling

• Define and scale standardized AI evaluation patterns and reusable quality frameworks acrossVLabs

• Contribute to enterprise AI quality standards and referencearchitectures

Governance & Responsible AI

• Ensure adherence to Responsible AI, data privacy, and governance requirements

• Support auditability, traceability, and transparency of AI outputs and evaluation processes

Stakeholder Enablement

• Translate evaluation results into actionable insights for engineering, product, and business stakeholders

• Support decision-making on model readiness, release risk, and performance trade-offs

• Proactivelyidentifyrisks, patterns, and systemic issues and escalate appropriately

Qualifications & Experience

Professional Experience

• 7+ years in software testing, including 2–3 years focused on AI/ML-enabled systems in production environments

• Proven experience designing and executing AI evaluation frameworks and quality strategies

• Strongtrack recordbuilding ground truth datasets, drift detection systems, and scalable evaluation pipelines

• Experience testing multi-step agentic workflows and AI-driven automation systems

• Experienceoperatingin fast-paced, iterative delivery environments

• Background in regulated or compliance-driven environments preferred

Technical Expertise

• Advanced Python programming for evaluation frameworks, batch processing, and data analysis

• Experience with LLM evaluation tools such asdeepeval, RAGAS,promptfoo, or similar

• Strong capabilities in:

• AI output validation, hallucination detection, and grounding checks

• Drift detection frameworks and statistical evaluation methods

• OCR, VLM, and document AI testing (classification, extraction, edge cases)

• API testing using Python (requests/httpx) and Postman/Newman

• SQL for data validation and pipeline integrity checks

• Familiarity withLangChain,LlamaIndex, or similar frameworks

• Experience with cloud AI platforms such as Azure AI Foundry or AWS Bedrock preferred

Operating Capabilities

• Ability tooperateindependently in fast-moving, ambiguous environments

• Strong analytical mindset with attention to detail and quality rigor

• Ability to balance speed and rigor in AI evaluation and delivery cycles

• Proactive communicator whoidentifiesrisks and drives resolution

• Ability to translate technical findings into business-relevant insights

Education

• Bachelor’s degreerequired;Advanced degree in Computer Science, Data Science, or related field preferred

We are an equal opportunity employer that does not discriminate on the basis of any legally protected status.

Please note, AI is used as part of the application process.