Spektra Systems

AI QA Engineer – Senior

Spektra Systems  •  Remote  •  3 months ago
Apply
AI can make mistakes so check important info. Chat history is never stored.
74
AI Success™

Job Description


This is a remote position.


About the Role


Traditional QA doesn’t work for AI.

We’re looking for a

Senior AI QA Engineer

who understands that AI systems are

probabilistic, non-deterministic, and failure-prone by nature

— and knows how to test them anyway.

You will own

AI quality, safety, reliability, and regression testing

across agentic systems and AI-powered SaaS products.


What You’ll Be Doing


AI-Specific Testing & Validation


  • Design and execute

    AI-specific test strategies

    , including:

    • Prompt robustness testing

    • Hallucination detection

    • Output consistency checks

    • Edge-case and adversarial testing

  • Validate

    RAG pipelines

    :

    • Retrieval accuracy

    • Context relevance

    • Response grounding

  • Test multi-agent workflows and tool integrations


Automation & Tooling


  • Build

    automated AI test harnesses

  • Create evaluation pipelines for AI responses

  • Define regression tests for prompt and agent changes

  • Integrate AI testing into CI/CD pipelines


Collaboration & Quality Ownership


  • Work closely with AI developers during design, not just after implementation

  • Help define acceptance criteria for AI features

  • Monitor AI behavior in production and flag drift or degradation

  • Document AI failure modes and mitigations


Requirements


Must Have


  • 4+ years of QA or testing experience

  • Strong API testing and automation skills

  • Understanding of AI/LLM behavior and limitations

  • Experience testing non-deterministic systems

  • Strong analytical and problem-solving mindset


Nice to Have


  • Experience testing AI agents or RAG systems

  • Experience writing Python-based test automation

  • Familiarity with LLM APIs and prompt engineering

  • Experience with observability tools for AI systems


Spektra Systems

About Spektra Systems

Spektra Systems is an innovation leader in cloud computing products and services. Our mission is to enable people and businesses to achieve more with the help of effective technological solutions.

Our four revolutionizing products are designed to help Cloud partners run and grow their businesses efficiently.

a) CSP Control Center (www.cspcontrolcenter.com)

Automating CSP business. Enabling growth.

C3 streamlines the entire business & technical lifecycle for Microsoft CSP partners. The platform provides an effortless self-service experience that scales productivity and business development.

b) CloudLabs (https://cloudlabs.ai/)

Transforming virtual learning through immersive technology.

CloudLabs is a self-service, end-to-end learning experience platform that delivers Microsoft Azure hands-on workshops at scale. Designed for training providers, ISV’s and system integrators, Cloudlabs has transformed the way cloud workshops and demonstrations are conducted.

c) SaaSify (https://saasify.ai/)

Helping ISVs and SaaS companies grow their business.

SaaSify is a one-of-its-kind solution built to simplify, streamline and accelerate revenue growth on cloud marketplaces. The platform enables companies to optimize, list, sell and manage their software offerings — cost-effectively.

d) Spektra Academy (https://spektraacademy.com/)

Empowering individual learners through hands-on lab training and certification.

Along with advanced solutions, we offer end-to-end services comprising advisory and consulting, product customization, and managed services.

We’re headquartered in Seattle, the United States with offices in Canada, India, and Mexico and clients worldwide.

Check us out at www.spektrasystems.com

Industry
IT & Software
Company Size
201-500 employees
Headquarters
Redmond, WA
Year Founded
Unknown
Social Media