Can you guys take please take a look at this agent agent_a6bf28cddd5a60502fc280fbce. We’re consistently getting a very slow response on the first real LLM-generated agent turn, e.g. see this call call_fa8313aece77de9992a3ced0582:
The STT + LLM + STT pipeline seems to have run in ~1 sec but the overall e2e time is ~4.2 sec. This happens 100% of the time for this bot (across a few dozen tests) and regardless of what TTS provider is chosen.
Hello @plamen from the looks of it, on this agent that residual is driven by the custom STT path (AssemblyAI streaming) + noise-cancellation denoiser + endpointingMs=300. Turn 1 pays the session/warm-up cost once and every later turn drops back to baseline.
Please try the following:
Switch sttMode from custom (AssemblyAI) to the default managed Deepgram STT — kills the custom-provider first-utterance warm-up.
Lower customSttConfig.endpointingMs (300 → 100–150) if you’re willing to trade some transcript accuracy for a faster commit.
Turn off denoisingMode: noise-cancellation if the input is already clean — removes first-frame denoiser cost.
LLM (gpt-4.1 HP cascading, TTFT ~700ms on a 4.3k-token prompt) and TTS (Cartesia sonic-3.5 ~80ms) look good, so nothing to action on them.
@Shah-Fazal switching to Deepgram helped but why are we getting this “warm-up” issue with AssemblyAI and not Deepgram? Which model from assemblyAI is used on the backend?
This seems to only happen when the customer says a specific first sentence E.g. if you answer the name question with “Hi, I’m John” this consistently results in 3-4 seconds first turn, if you just say “John”, it’s consistently ~1sec. It does only happen with AssemblyAI though - do you use AssemblyAI’s turn-taking decisions instead of your turn-taking model or a combination or both? Also you use universal-3.5-pro as the STT model, right?
Here are some examples.
Answer only with the name: