Job Description

Position Overview

The Software development engineer in test is responsible for ensuring the quality, reliability, and performance of AI agents and agentic experiences. This role involves designing and executing test strategies, identifying defects, and working closely with engineering teams to ensure high‑quality releases of AI agent solutions for business use cases.

This role requires a strong attention to detail, analytical thinking, and the ability to think like both a user and a developer. The ideal candidate is passionate about quality and enjoys finding creative ways to test AI agents and ensure they work correctly across various scenarios and edge cases.

The ideal candidate has experience testing AI/ML applications, with particular strength in testing conversational interfaces, LLM integrations, and AI agent workflows. This person should also have the desire to grow and learn everyday which will be essential for success in this role. The landscape changes daily and we are changing with it.

Key Responsibilities

• Design and execute test plans for AI agents and agentic experiences.
• Write and maintain automated test suites for agent functionality (unit tests, evals integration tests, end‑to‑end tests).
• Perform (minimal)manual testing of agent interactions, workflows, and business logic.
• Test agent responses, accuracy, and behavior across various scenarios and edge cases.
• Identify, document, and track bugs through resolution.
• Collaborate with engineers, product managers, and business stakeholders to understand requirements and acceptance criteria.
• Participate in test planning, test case design, and test strategy discussions.
• Create and maintain test data, test scenarios, and test environments for agents.
• Participate in feature design sessions, highlighting key testing scenarios and fault zones.
• Execute performance and load testing to ensure agent scalability and response times.
• Validate agent integrations with business systems, APIs, and data sources.
• Test agent security features and validate compliance with security requirements.
• Participate in release planning and ensure quality gates are met before releases.
• Contribute to improving testing processes and test automation infrastructure for AI agents.

Required Qualifications

• 3+ years of experience in software quality assurance or testing.
• 1+ years of experience testing AI/ML applications, LLM integrations, or conversational interfaces.
• Strong understanding of software testing principles, methodologies, and best practices.
• Experience writing and maintaining automated tests (unit, integration, or end‑to‑end).
• Proficiency in at least one programming language (Python, TypeScript, JavaScript, Java, etc.).
• Experience with API testing tools (Postman, REST Assured, etc.) or frameworks.
• Strong analytical and problem‑solving skills.
• Excellent attention to detail and ability to identify edge cases.
• Good written and verbal communication skills.
• Experience with bug tracking systems and test management tools.
• Ability to work collaboratively with engineering and product teams.
• Understanding of CI/CD pipelines and test automation in continuous integration.
• Interest in AI/ML concepts and understanding of how to test AI systems.

Preferred Qualifications

• Experience with AI Eval tools or frameworks.
• Experience testing AI agents, chatbots, or virtual assistants.
• Background in testing LLM integrations and prompt‑based systems.
• Experience with agent testing frameworks and tools.
• Knowledge of testing RAG (Retrieval Augmented Generation) systems.
• Experience with performance testing tools (JMeter, k6, Locust, etc.).
• Experience with test automation frameworks (Playwright, Cypress, Selenium, pytest, etc.).
• Familiarity with cloud platforms and testing cloud‑native applications.
• Experience with observability tools and using metrics/logs for test validation.
• Knowledge of security testing and vulnerability assessment for AI applications.
• Experience with contract testing and API mocking.
• Familiarity with prompt testing and LLM response validation.
• ISTQB or similar testing certification.
• Chaos and Resilance Testing

Why This Role Matters

Quality Assurance Engineers are essential to ensuring AI agents meet the high standards required for business use. By finding defects early and ensuring comprehensive test coverage, QA Engineers prevent issues from reaching production and protect the reliability of agentic experiences. This role offers the opportunity to work across the entire agent development lifecycle, understanding how agents integrate with business systems and ensuring they work correctly end‑to‑end. If you enjoy solving puzzles, finding edge cases, and ensuring AI systems work perfectly for real users, this role provides the opportunity to make a significant impact on the quality and success of agentic experiences.