CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our nearshore technology services empower businesses to thrive in an ever-evolving digital landscape.
We are looking for a Junior AI Agent Quality Engineer to help us build and validate the next generation of autonomous systems. In this role, you won't just be checking for bugs; you'll be evaluating the "brain" of our AI.
You will focus on the reliability of Agentic AI and RAG (Retrieval-Augmented Generation) You’ll be responsible for ensuring our agents can plan tasks, use tools accurately, and recover from errors without "hallucinating." This is a perfect role for someone with a QA mindset who is passionate about Python and the future of LLM-based orchestration.
Agent Evaluation: Create question templates and Python scripts to test how well AI Agents and RAG instances solve complex tasks.
KPI Tracking: Evaluate Agent performance using specific metrics: Success Rate, Tool Use Accuracy, Planning Quality, and Autonomy
Dataset Creation: Collaborate with AI Engineers to build "Ground Truth" datasets and Agentic Task Datasets to benchmark model improvements.
API & Logic Testing: Write unit and integration tests for Python-based RESTful APIs and agent endpoints.
Performance Testing: Run load and bulk testing (using Locust or JMeter) to see how our AI handles high-volume requests.
Input/Output Validation: Perform rigorous testing of agent payloads to catch prompt injection risks and logic failures.
Agentic AI Focus: Basic understanding of Agentic workflows, prompt engineering (ReAct prompts), and LLM orchestration.
AI Observability: Familiarity with (or a strong desire to learn) tools like LangSmith, Langfuse, or OpenTelemetry
Python Skills: Proficiency in Python 3.10+ for automation and data manipulation.
API Fundamentals: Strong experience testing and validating RESTful APIs and JSON structures.
Analytical Thinking: A "breaker" mindset—the ability to find edge cases where an AI might fail to follow instructions.
Language: Advanced English (B2/C1) for global team collaboration and technical documentation.
QA Experience: 1–3 years of experience in Software QA (Manual or Automation).
Automation Tooling: Exposure to Pytest, Playwright, or Selenium.
AI Frameworks: Exposure to LangChain, LangGraph, or Pydantic AI
Infrastructure: Basic knowledge of Docker and Vector Databases.
Performance: Experience with Locust or JMeter for bulk testing.
At CodeRoad, we accelerate ROI through end-to-end software engineering and technology execution.
Our Velocity-as-a-Service (VaaS) delivery engine deploys elite nearshore delivery pods and AI-ready systems that execute your technology roadmap, deliver measurable outcomes, and scale innovation with certainty.
Founded in 2002, CodeRoad pioneered nearshore engineering delivery from La Paz, Bolivia. Over two decades, we have expanded across Latin America to build a curated network of senior engineers, product leaders, and AI specialists who collaborate in real time with North American technology teams.
Through senior-led delivery pods and mature engineering frameworks, we help organizations move from strategy to execution—turning complex technology initiatives into production-ready systems without the delays that slow innovation.
Our teams support companies across industries including retail, automotive, transportation and logistics, food and beverage, financial services, and energy.
We operate across the modern technology landscape, mastering languages, frameworks, cloud platforms, and AI-enabled development environments to support next-generation digital systems.
With Velocity-as-a-Service, organizations gain the execution architecture needed to launch products faster, modernize platforms, and drive digital transformation with measurable ROI.
CodeRoad gets you there — at full velocity.
