Any retell-llm with an s2s_model produces calls where general_prompt is clearly never injected: the agent responds like a raw unprompted model (“Hi there! How can I help today?”) with zero awareness of its saved ~1,700-token prompt.
What works vs what doesn’t (same LLM, same version, same prompt):
Cascading model: gpt-5.4-mini → follows the prompt perfectly (simulation + web calls)
s2s_model → blank generic assistant, reproduced in 4 dashboard Test Audio web calls across bothgpt-realtime-mini and gpt-realtime-2.1-mini, both user-first (begin_message: "") and agent-first (begin_message: "Hello?")
The rest of the config IS honoured (silent start / begin message play correctly), and GET /get-retell-llm confirms the prompt is saved on the tested version.
Config at failure: 9 general_tools (custom webhook tools, tool_call_strict_mode: true), 1 knowledge base, no states.
Ask: can you check what instructions the realtime session actually received for those calls? Is there a known issue with the S2S bridge dropping instructions (failed session.update from strict-mode tool schemas, context packing, etc.)?
Hey @willbird1936 We confirmed your general prompt is being sent correctly.
The issue is specific to Realtime S2S handling of the first turn, especially when the opening audio is ambiguous or the prompt expects the agent to remain silent.
Until we fix this:
1.For reliable silence/IVR handling, switch to a cascading model such as GPT‑5.4 mini. Your existing NO_RESPONSE_NEEDED instructions should work there.
2.If you need to remain on S2S, use agent-first mode with the complete intended opening as the begin message—not just “Hello?” or a silent/listening turn.
3.Don’t rely on NO_RESPONSE_NEEDED in S2S for now; S2S doesn’t currently have an equivalent reliable silent action.
4.Keep the call-start rules short and place them at the very beginning of the prompt, but note that this only mitigates the issue—it doesn’t fully resolve it.
If detecting a human versus voicemail/IVR before speaking is required, cascading is currently the recommended option.
Hey @willbird1936 Team took a further look into this issue, repro’d it and found the following:
What it is not. Your general_prompt is being delivered to the realtime model correctly and in full. We verified this at the API level: on every S2S turn the model’s instructions contain your complete prompt, and we confirmed the realtime session acknowledges storing it. There is no bridge bug dropping instructions, and strict-mode tool schemas are not the cause.
What it is. We believe this is just hallucination on the mini realtime models (gpt-realtime-mini, gpt-realtime-2.1-mini), specifically on the first turn of a “user speaks first” call.
In my testing, the mini model followed a sample ~1800 token prompt only about 1 in 5 cold-start calls. The full-size model (gpt-realtime-2.1) followed the prompt reliably in the same conditions.
Reccomendations:
As asserted earlier, if detecting a human versus voicemail/IVR before speaking is required, cascading is currently the recommended option.
If you want to use S2S, consider using the full realtime model gpt-realtime-2.1, it handles complex system instructions better.
If you must use the mini, consider putting the more important directions towards the bottom of the system prompt. In my testing, I moved the “first message” instruction to the very last and got the mini model working reliably.