AssemblyAI STT high E2E latency even though STT latency is low

Support Help
1

Hello,

We’re noticing a couple of weird behaviors around AssemblyAI STT and the E2E latency between turns:

  1. When dynamic responsiveness is set to true and STT is AssemblyAI, the turn takes up all the time given in the “endpointing” config. For example, see these calls call_40cb7cc755a20702670e9cd4276, call_6b53241aaf93a97504fa3605869, call_83f6923849fae1a5e2c2ba5b318 and call_549fa8895b969d9862f76ef8500. The median E2E latency times in all of those are upwards of 3000ms but if you see the individual components, they don’t add up at all: STT median 239 ms, LLM median 783 ms, TTS median 183 ms. Seems we’ve lost 2 seconds somewhere :slight_smile:

  2. Everything the same but without dynamic responsiveness (and responsiveness set to 1), things look better but still E2E latencies are very high e.g. call_c254dc13c87d71d135cac450a71. Again, the separate components don’t add up at all.

And here are a few more examples with endpointing = 500ms where we still get average E2E times of 1800ms where the individual components take much less: call_f6c812e7e856ee48814d40ba8bf, call_84fd2a164d0e2106a1a6f8bd50c. These were somewhat of outlier though, and in general with these settings the E2E time was better. Still curious where all that time is going.

Can you guys investigate where that E2E latency time is lost (if not for the 3 main components - STT, LLM, TTS) - we’re talking about seconds here which makes or break the flow of a phone call. Anything we can do about it?