Hey, am resimulating call using the retell batch simulations feature.
knowledgeBaseRetrievedContents is pulling the wrong chunks during batch simulations but looks like working fine when testing it normally (in “Test agent” → “test llm” → “Manual Chat”).
If we can’t test the knowledge base retrieval, the knowledge base feature is literally useless. I saw that someone else raised this question in discord (in 2024!) and it’s still not fixed.
How to fix this?
Hey Liana — we’ve run into similar issues, and for production RAG agents we still manually test large call sets rather than relying only on batch simulations.
Batch simulations are useful for surface-level regression testing, but RAG evaluation is more sensitive because retrieval depends on the exact query phrasing, conversation state, timing, transcript normalization, and how the simulation runtime packages context into the LLM call. If the batch simulation environment is not behaving exactly like the live/test-agent runtime, you can end up debugging the simulator rather than the actual agent.
In our experience, manual RAG testing leaves much less room for false confidence or false negatives. We test hundreds of realistic call paths, check which chunks are retrieved, adjust the KB structure, and then retest against real caller phrasing. It takes longer upfront, but it gives a much clearer picture of whether the agent will actually work in production.
I still think Retell has a lot more upside for RAG agents than most people realize. The KB capability is stronger than it looks when the content is structured properly and tested against realistic conversations.
So I agree the batch simulation will likely evolve over time, but I wouldn’t write off the KB feature because of that. A working production agent with manually validated retrieval is much more valuable than a clean batch test that may not reflect real behavior.
One of our largest projects is leveraging an extremely sophisticated KB retrieval system that only Retell is currently able to handle.