Job Description

We are seeking an AI Quality & Content Specialistto ensure the accuracy, reliability, and continuous improvement of our AI support experience. This role will act as the qualitygatekeeper;identifyingrisks, improving performance, and building trust in AI across both customers and internal teams.

This person will also own the ongoing auditing and governance of the knowledge and source content powering AI experiences to ensure outputs remain accurate, trustworthy, and operationally effective.

Key Responsibilities

AI Quality Assurance

• Audit AI-generated responses across:

  • Chat

  • Cases

  • Self-service flows

• Identify:

  • Hallucinations and reduce “assumed resolves”

  • Incomplete answers

  • Compliance risks

• Score AI performance against defined QA frameworks

• Audit and validate the knowledge content, workflows, and source materials leveraged by AI systems to ensure responses remain accurate, comprehensive, compliant, and aligned with current product functionality and support policies.

• Identify outdated, conflicting, incomplete, or low-confidence content that may negatively impact AI-generated responses and customer experience.

• Establish ongoing content governance processes to maintain AI readiness across help center articles, macros, internal documentation, and workflow logic.

Root Cause Analysis

• Analyze why AI fails:

  • Content gaps

  • Poor prompt handling

  • Edge cases

• Provide actionable insights to:

  • Content Strategist

  • Head of Support

• Create a feedback loop with Product and Engineering and serve as the voice of the customer

Performance Reporting

• Build and maintain dashboards tracking:

  • Resolution accuracy

  • Escalation drivers

  • Customer sentiment (AI vs human)

• Surface trends and recommend improvements

Continuous Improvement Loop

• Partner with stakeholders to:

  • Retrain AI

  • Improve knowledge content

  • Adjust workflows and guardrails

• Help establish a closed-loop AI optimization system

• Partner with Content, Product, Engineering, and Support stakeholders to continuously optimize and maintain AI source content quality.

• Create scalable auditing mechanisms to proactively identify content gaps, stale documentation, and areas where AI confidence or resolution quality declines.

• Help define standards for AI-consumable knowledge design to improve answer consistency, containment, and customer trust.

Success Metrics (First 12 Months)

• Reduction in AI error rate / false positives

• Improved AI CSAT and containment rate

• Increased AI trust score

• Increased adoption of the AI channel

• Faster identification of content gaps

• Established QA framework for AI governance

Where: Remote, US
TheEverCommerceteam is distributed globally, with teams inthe U.S., Canada, the U.K., Jordan, New Zealand, and Australia With a widely distributed team, we are used to working remotely across different time zones. This role can be based anywhere in the United States– ifyou’reclose to one of our offices, we can set you up in-office or you can work 100% remotely. Please note that you must be eligible to work without sponsorship to qualify for this position, and this role may require travel to our Corporate Headquarters in Denver, Colorado, or toother office locations around North America.