Lesson 22: AI QA & Simulation Testing

Retell University Module 6: Monitor, Test & Optimize
1

AI QA & Simulation Testing

Simulation Testing

Test your agent at scale without making real calls:

  • Define test scenarios with expected behaviors
  • Run hundreds of simulated conversations
  • Validate that the agent handles each scenario correctly
  • Catch regressions before deploying changes

Setting Up Simulations

  1. Go to Testing → Simulation
  2. Create test cases:
    • Caller persona (who is calling and why)
    • Expected agent behavior
    • Success/failure criteria
  3. Run the simulation batch
  4. Review results — pass/fail for each scenario

AI QA

Automatic quality assurance across your live calls:

  • Define cohorts — groups of calls to evaluate
  • Set metrics — what does a good call look like?
  • Resolution criteria — was the caller’s issue resolved?
  • Get aggregate scores and identify problem areas

QA Workflow

  1. Set up cohorts (e.g., all calls this week, calls from agent X)
  2. Define evaluation criteria
  3. Review QA dashboard for scores and trends
  4. Drill into low-scoring calls
  5. Update prompts/flows and re-test

Agent Versioning

  • Every change to your agent creates a new version
  • Compare performance across versions
  • Roll back to a previous version if a change causes issues
  • A/B test different versions