Hi,
So in the automated tests section of the agent it gives an error 500 for all the tools I have set up in the agent, However when I manually test it, the tool runs fine.
Why do our custom function/tool calls work in manual Test Chat but consistently return errors in Simulation/Batch Testing? Is Simulation using a different execution path, environment, or request format for tool calls?
Is Retell Simulation intended to support testing real live tool calls end-to-end, or is it mainly designed for mocked function outputs and agent-response testing? If live tool testing is supported, what exact setup is required to make it work reliably.
Simulation testing supports function mocks specifically to avoid this problem. You can “set up mocks for custom functions to prevent calling actual functions during testing, ensuring consistent results across test runs.”
This strongly suggests that simulation testing may not reliably execute live tool calls the same way manual testing does — the recommended approach is to use function mocks for consistent results. The 500 errors you’re seeing likely occur because the simulation environment handles tool call execution differently (e.g., different request context, missing call metadata, or different authentication headers sent to your endpoint).
What to do:
Configure function mocks in your test cases to return expected responses instead of hitting your live endpoints.
If you need true end-to-end live tool testing, use manual test chat or phone call testing instead.
If still this happens you can share your Agent Id with us.
Thanks @shaw . I understand mocks are the recommended workaround, but my goal here is different: I want to use Simulation/Batch Testing as a recurring regression check for whether our live tools are actually working end-to-end. something like making sure product is working as intended(including testing if the functions are working correctly, setting up mock results dosent address this)
Right now, the same tools work correctly in Manual Test Chat, but fail consistently in Simulation/Batch Testing with 500 errors. So I’m trying to confirm whether live custom function/tool execution is officially supported in Simulation at all.
Could you clarify:
Is Simulation/Batch Testing expected to support real live tool calls end-to-end, or are live tool calls not fully supported there today?
If live tool calls are supported, what is the exact required setup so Simulation sends the same request context as Manual Test Chat?
If they are not supported, is there any recommended way to run scheduled/periodic automated tests that validate actual tool health instead of mocked outputs?
