Job Title: Senior QA & Evaluation Engineer
Key Skills: Test Automation, .NET, API Testing, CI/CD, Automated Testing, LLM Evaluation, AI Testing, Test Frameworks
Experience: 4+ years of experience.
Location: Costa Rica, Peru, Colombia, Bolivia.
Mode: Remote.
We at Coforge are hiring Mid-Senior QA & Evaluation Engineer (N-5) with the following skill set.
· Design and maintain automated test suites covering unit, integration, API, and end-to-end testing.
· Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
· Build evaluation frameworks for AI capabilities, including scored rubrics, LLM-as-a-judge approaches, and normalized pass thresholds.
· Build and maintain held-out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results as part of delivery metrics.
· 4+ years of experience in Quality Engineering with a strong focus on test automation.
· Hands-on experience with automated testing frameworks, preferably within the .NET ecosystem.
· Strong experience with API testing and automated test development.
· Experience integrating automated test suites into CI/CD pipelines.
· Understanding of testing strategies for non-deterministic and probabilistic AI systems.
· Ability to design evaluation approaches based on scored thresholds rather than exact-match assertions.
· Experience working with test data, ground truth datasets, and automated validation.
· Strong analytical and problem-solving skills.
· Strong working proficiency in English.
· Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.
· Direct experience with LLM evaluation, including rubric design, LLM-as-a-judge, golden/held-out datasets, or tools such as Azure AI Foundry Evaluators, promptfoo, Ragas, or DeepEval.
· Experience testing information extraction or classification pipelines using accuracy-based measurements.
· Experience with regulated-industry QA environments where test evidence and results must support formal review.
· Familiarity with AI/ML testing methodologies and evaluation frameworks.
Posted On: 24-09-2026
At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
Encora is a global digital engineering company built for a new era of AI-powered businesses. Forged in Silicon Valley, we grew up serving digital natives who demanded speed, scale, and flawless experiences. Today, we bring that same DNA to enterprises worldwide, enabling speed and hyper-adaptability, moving them further along their quest to become more composable and agile.
With two decades at the convergence of Cloud, Data, and AI, we help organizations move beyond rigid, legacy models to become composable enterprises, through adaptive systems designed to sense change and reconfigure in real time.
More than 9,500 engineers, designers, and domain experts across the Americas, Europe, India, and Southeast Asia give Encora the ability to combine nearshore agility with global scale. Backed by Advent International and Warburg Pincus, we deliver innovation acceleration and modernization across industries, powered by Encora AIVA™, our orchestration platform for AI-native engineering.
We believe the future won’t be built by humans or machines alone, but by human creativity amplified by limitless machine intelligence. At Encora, that’s how we help our clients build fast and adapt faster.
