We’re noticing a couple of weird behaviors around AssemblyAI STT and the E2E latency between turns:
When dynamic responsiveness is set to true and STT is AssemblyAI, the turn takes up all the time given in the “endpointing” config. For example, see these calls call_40cb7cc755a20702670e9cd4276, call_6b53241aaf93a97504fa3605869, call_83f6923849fae1a5e2c2ba5b318 and call_549fa8895b969d9862f76ef8500. The median E2E latency times in all of those are upwards of 3000ms but if you see the individual components, they don’t add up at all: STT median 239 ms, LLM median 783 ms, TTS median 183 ms. Seems we’ve lost 2 seconds somewhere
Everything the same but without dynamic responsiveness (and responsiveness set to 1), things look better but still E2E latencies are very high e.g. call_c254dc13c87d71d135cac450a71. Again, the separate components don’t add up at all.
And here are a few more examples with endpointing = 500ms where we still get average E2E times of 1800ms where the individual components take much less: call_f6c812e7e856ee48814d40ba8bf, call_84fd2a164d0e2106a1a6f8bd50c. These were somewhat of outlier though, and in general with these settings the E2E time was better. Still curious where all that time is going.
Can you guys investigate where that E2E latency time is lost (if not for the 3 main components - STT, LLM, TTS) - we’re talking about seconds here which makes or break the flow of a phone call. Anything we can do about it?
Here is another example with endpoiting_ms=3000, dynamic_responsiveness=false:
2026-07-29 07:16:03.412 call_b0682c40c1cbec7b294faa25b73 info: Latency for response_id: 1 - e2e: 4907 asr: 285 llm: 1435 tts: 156
Hello @plamen
The unaccounted 2–3s is almost entirely the end-of-turn silence wait — e2e is measured from when the caller stops speaking to the first agent audio byte, so it includes endpointing/VAD hold; asr, llm, tts are inference-only sub-timers and by design won’t sum to e2e.
call_b0682: residual ≈ 2931 ms lines up exactly with the endpointing_ms=3000 you had set on that run — the endpointer waited the full window on a trailing utterance before committing the turn.
call_40cb7c: with endpointing_ms=300 the ~2s residual is larger than the configured floor alone; the segmented user utterances (“Just a tub.”, “fiberglass.”) plus interruption_sensitivity=0.9 keep VAD accumulating through short silences before the turn commits.
To shrink the wait predictably: keep responsiveness=1 and endpointing_ms in the 300–700 range, set dynamic_responsiveness=false, and try lowering interruption_sensitivity toward ~0.7 on a test agent to see if the residual drops (this primarily controls barge-in, but at 0.9 it also lengthens the listen window on choppy audio).
Thanks a lot for the response. I need a bit more of your help to fully understand what’s going on here:
endpointing_ms - can you explain what this is and how Retell uses it? Is this used by Retell alone or also passed to the STT/ASR providers (as they all do endpointing also)?
E2E latency - can you explain when precisely does the clock for this start ticking. Is it when the STT/ASR provider declares the customer turn transcription as final or earlier? Other than that it seems it also includes the LLM latency + the TTS latency to first byte, right?
ASR/STT latency - is this the time from the first byte sent to the STT/ASR provider until the time to final(non-partial) transcription?
TTS latency - is that the time to first byte from the TTS model or is it the time needed to fully convert the text to audio?
Endpointing_ms — the silence window Retell waits after you stop speaking before committing your turn to the LLM. It’s used both by Retell (to close the turn) and passed through to the STT provider as their turn-silence / endpointing parameter, so the STT and Retell agree on when your utterance ends. Longer values → more accurate transcripts, but they add directly to e2e. e2e — the clock starts when the user stops speaking (end of user audio) and stops when the first agent audio byte is produced. So yes, it includes the endpointing/VAD hold + ASR finalization + LLM time-to-first-speakable-chunk + TTS time-to-first-byte. It does not include network trip time from Retell to the user’s device. asr — this is a transcription-lag metric, not a request/response wall clock: the difference between the duration of audio we’ve streamed to the STT and the duration of audio that’s been transcribed back. It does not include the endpointing wait. tts — time to first audio byte from the TTS model (trigger → first byte), not full text-to-audio synthesis time. Because asr, llm, tts are inference-only sub-timers and e2e also includes the endpointing wait, the components will never sum to e2e — the endpointing/VAD hold is the delta you’re seeing. Docs: Check actual latency - Retell AI and Troubleshoot high latency - Retell AI .
Thanks for the info. Things still don’t add up though and there are some unexplained delays - can you please take a look at this call: call_5d193e824ae081e1efdc9cb750f
This is the config it was run with:
stt=assemblyai endpointing_ms=3000 (we don’t intend to run with such high values but it’s easier for illustration) responsiveness=1 enable_dynamic_responsiveness=false interruption_sensitivity=0.8
My understanding (per your last comment) - endpoiting_ms is a flat time window that is always awaited after a user stops speaking and before the turn is committed to the LLM. This value is also passed as min_turn_silence to AssemblyAI - correct? - and what is passed as max_turn_silence or is it the default 1536 per Turn Detection - AssemblyAI?
Looking at the details log of call_5d193e824ae081e1efdc9cb750f we see this for the first turn:2026-07-31 17:13:39.667 call_5d193e824ae081e1efdc9cb750f info: Latency for response_id: 1 - e2e: 4509 asr: 237 llm: 871 tts: 173
So here is how this should look based on what you explained:
time ─────────────────────────────────────────────────────────────────────▶
caller stops first agent
speaking audio
│ │
CALLER ██████████│· · · · · · · · · (silence) · · · · · · · · · │
│ │
ASR ▒▒▒▒▒▒▒▒▒▒│██▌ │
│ └─ ASR TAIL ~237 ms — still transcribing the │
│ last words after speech ends; finishes │
│ inside endpointing window, adds 0 to e2e │
│ │
ENDPOINTING │░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
│◀───────── 3000 ms ──────────▶ │
│ starts the moment speech stops; │
│ │
│ │
??? │ ▐███▌ ~465 ms │
│ └─ window expiry →
│ LLM start; what
│ happens in here?
│ │
LLM │ █████████ │ 871 ms
│ │
TTS │ ██│ 173 ms
│ │
AGENT │ ▐███████▶
│ │
│◀═══════════════ e2e 4509 ms ════════════════▶│
Even with the 3000ms endpointing_ms there is 465ms (marked as ??? on the graphic) - what is happening here? We see such a residual (between 200-400ms) even with lower endpointing values (e.g. here call_1c3064bb1ebeaca48d13ed2c614 with endpoiting_ms=300). This is a lot of time that really can break the call experience.
And here is the second turn from the same call:2026-07-31 17:13:57.644 call_5d193e824ae081e1efdc9cb750f info: Latency for response_id: 2 - e2e: 1332 asr: 285 llm: 583 tts: 201 This seems like a bug in the e2e latency reporting - clearly the e2e time can’t be 1332 since only endpointing_ms is 3000 - also if you listen to the actual recording you can see that there is a ~4sec gap at the second turn, not 1 sec as reported.
Finally, where does the responsiveness parameter play a role in all of this.
