STT transcript rollback causes repeated LLM restarts on first caller utterance (Soniox, fr-FR)

Support Help
1

Hello,

We run a French-language inbound voice agent on Retell and see a reproducible 2.5–4 second delay on the caller’s FIRST utterance. It occurs in roughly half of inbound calls. Later turns in the same call are consistently fast (1.2–1.4 s).

Agent: agent_9e0dee3ccc3bb725e099d78121

Language: fr-FR | Conversation Flow, Rigid Mode, GPT-4.1

STT: custom, provider Soniox

Telephony: Twilio Elastic SIP Trunking

WHAT THE DETAIL LOGS SHOW

The delay correlates exactly with the number of “llm input token length calculated” entries before the first “Latency” line. Each extra entry adds roughly 1.5 s to e2e.

Fast call — call_aff87aeee065248f95337d34b30

08:57:58.876 tokenLength 3855

08:57:59.296 tokenLength 3858 (+420 ms)

08:57:59.341 e2e 1194

Slow call — call_97ae094b51479567a7f3e00cd20

10:13:14.563 tokenLength 3464

10:13:16.090 tokenLength 3472 (+1527 ms)

10:13:17.686 tokenLength 3472 (+1596 ms, identical token count)

10:13:18.352 e2e 2767

Slow call — call_f69c38296b501f44a5e931c14e3

10:27:13.897 tokenLength 3513

10:27:13.946 tokenLength 3512 (+49 ms, token count DECREASED)

10:27:15.870 tokenLength 3521

10:27:18.620 tokenLength 3523

10:27:19.054 e2e 3776

Two observations we cannot explain:

  1. In the third call the token count DROPPED between runs, 49 ms apart. This looks like the STT retracting text it had already emitted, causing Retell to discard the in-flight generation and restart.

  2. In the second call one full ~1.6 s cycle produced an identical token count, so the restart added no new information at all.

Affected calls also contain junk tokens at the end of the transcript — a trailing “et”, “,.” or “dans.” that the caller never said. Junk token and delay always appear together; we have 17 calls with no exception.

WHAT WE ALREADY TESTED (no effect)

• endpointing_ms 840 / 1000 / 1200 — 1200 made it worse (4 runs instead of 3)

• shortening node edge conditions and global prompt

• ruling out overlap with the agent’s greeting (caller waited for it to finish)

• ruling out cold start (first call after an hour of silence was the fastest)

QUESTIONS

  1. Why does the transcript get retracted, and can interim-result stabilisation be configured?

  2. Is there a setting that makes Retell wait for a final transcript before starting generation?

  3. Does denoising_mode affect the audio sent to the STT provider?

  4. For French inbound telephony, which STT provider do you recommend? We chose Soniox because Deepgram could not recognise isolated French commune names.

Happy to provide additional call IDs or recordings.

Thank you

2

Switched to Deepgram — no LLM restarts, but asr latency on the first utterance jumped to 1167–2125 ms (vs 550–690 ms on later turns). Delay persists on both providers, so it appears to originate in pipeline warm-up rather than in the STT engine itself. Calls: call_23909512…9b5, call_144b01014…382, call_1f68051717…3df.

(Call IDs abbreviated above — happy to send the full IDs on request.)

3

Same delay on stt_mode: fast — call_cfd8c20efcc60fbcf5be2d3d01b, response_id 1: e2e 2961 with asr 342 + llm 485 + tts 67 = 894 ms, single LLM run. 2 seconds unaccounted for on the first utterance only; later turns in the same call show 177–540 ms of overhead.

4

Hello @Aleksandr

I have escalated your request to the relevant team. As soon as I receive any updates, I’ll let you know.

Best Regards.

5

Hello @Aleksandr

Thank you for the extra call IDs.

The stt_mode: fast call invalidates the earlier explanation that pointed only at transcript retraction, since that call ran a single LLM generation and was still slow. Here is the corrected read.

On call_cfd8c20efcc60fbcf5be2d3d01b the sub-timers are asr 342 ms, llm 485 ms and tts 67 ms, which totals 894 ms against an e2e of 2961 ms. The asr, llm and tts values measure inference only. They exclude the VAD and endpointing wait between the end of caller speech and the decision that the turn is complete, and they exclude ASR session warm-up (asrWebsocketOpen was 256 ms on that call). e2e includes both. That is why 894 ms of components fits inside 2961 ms, and why later turns in the same call carry only 177 to 540 ms of overhead: the ASR session is already open and the VAD is calibrated.

Two independent layers stack on the first utterance, not one.

Base first-turn cost, roughly 1.5 to 2 seconds, independent of which STT provider you use. This is the post-speech VAD wait governed by responsiveness, plus ASR session warm-up. This is the layer you see isolated on cfd8c20e, which had a single generation. One important detail: custom_stt_config.endpointing_ms is only applied in stt_mode: “custom”. Your 840 / 1000 / 1200 tests therefore had no effect once the agent was in fast mode, and in custom mode raising it to 1200 made things worse because it widened the window for an additional cycle of layer 2.

Transcript retraction, roughly 1.5 seconds per cycle, additive on top of layer 1. The most likely mechanism is that denoising_mode: noise-cancellation reshapes the trailing breath or room noise at the end of the first utterance into a revisable French token, which is what produces the phantom dans., ,. and et you see. Each retraction restarts the generation and produces one more llm input token length calculated entry. On call_97ae094b51479567a7f3e00cd20 the llmTtft array holds three distinct values (534 / 335 / 435 ms), which matches the three runs in your log, and the fast call aff87a has no phantom trailing token at all. Your two unexplained observations follow from this: an identical token count means the revised transcript tokenized to the same length, and a decreased token count means the revision removed text that had already been sent.

To your four questions, in order.

Why the retraction happens, and whether interim stabilization is configurable: the Soniox stream does not expose an is_final flag, so finality is inferred by diffing words between frames. There is no client-side interim stabilization control for that provider.

Whether generation can wait for a final transcript: there is no separate flag for this. The gate is responsiveness combined with VAD. Raising responsiveness shortens the wait. endpointing_ms, in custom mode only, lengthens it, which helps accuracy and hurts first-turn latency.

Whether denoising_mode affects the audio sent to the STT provider: yes. Denoising is applied before the audio reaches the provider, which is why it is the prime suspect for layer 2.

STT recommendation for French inbound telephony: before staying on custom Soniox, benchmark the built-in accurate transcription mode with language: “fr-FR” on your commune list. Accuracy mode uses a higher endpointing setting and is specifically better at capturing entities such as numbers, dates and proper nouns, which is the same class of problem as isolated commune names. The documented cost is about 200 ms of additional ASR latency.

Suggested test sequence, on a copy of the agent, run in this order since the steps are additive.

Keep your current stt_mode and raise responsiveness from 0.9 to 1.0. Expect roughly 500 ms to 1 s off the first turn, with no change to later turns.

On the Soniox copy, disable denoising_mode and run 5 to 10 calls. If the phantom token and the multiple tokenLength entries disappear, layer 2 is confirmed and the first turn should fall back to the layer 1 baseline. Please treat this as a diagnostic run rather than a permanent change until the result is confirmed, since a different root cause would need a different fix.

In parallel, test stt_mode: “accurate” with fr-FR on the same numbers to compare commune recognition against Soniox.

Send us 3 to 5 call IDs after each step and we will verify the breakdown in the logs.

Reference: Troubleshoot high latency - Retell AI and Balance between transcription accuracy and latency - Retell AI

Best Regards.

6

Same account, same Twilio number, same fr-FR language.
Agent A (fast): first turn e2e 1354 ms — asr 298 + llm 713 + tts 106 = 1117.
Overhead 237 ms. No stt_mode set, no denoising_mode set, single-node flow.
Agent B (slow): first turn e2e 3287-3360 ms across five calls, 73 ms spread.
Component sums never exceed 1250 ms. stt_mode custom (Soniox), multi-node flow.
If the first-turn VAD wait and ASR warm-up are provider-independent, both
agents should show the same baseline. Agent A does not.
We also ran five calls with denoising disabled: phantom trailing tokens mostly
disappeared, e2e unchanged. Layer 2 removed, layer 1 unaffected.
Call IDs available on request — transcripts contain personal data, I’d rather
share them privately.

7

responsiveness 1.0 made it worse: median e2e went from 1501 to 2410 ms
(call_79bbee7c40cc8294bc7…285). Reverting to 0.9.
We have now tested every setting you suggested plus endpointing 840/1000/1200,
Deepgram, fast mode, denoising on/off and handbook flags. None changed the
first-turn constant.
The only remaining difference between our slow agent and the fast one on the
same account is flow architecture: multi-node Rigid vs single-node Flex.
Does conversation flow structure affect first-turn latency in a way that would
not appear in the asr/llm/tts breakdown?

8

Hello @Aleksandr

Could you please provide the relevant Call IDs . This will help us look into the issue and investigate the call attempts further.

Thank you!