Error encountered in LLM call: Mid-stream error in stream: Operation timed out

Support Help
1

Posting this with data rather than a single call, because the pattern is what matters.

**What I measured**

I pulled the public log for all 3,258 calls in my workspace over the last 7 days and grepped for the error. 107 calls affected, 276 total occurrences.

Distribution by day:

| Day | Calls | Affected | Occurrences | Hit rate |

|—|—|—|—|—|

| Aug 20 | 571 | 0 | 0 | 0% |

| Aug 21 | 513 | 0 | 0 | 0% |

| Aug 22 | 375 | 1 | 1 | 0.3% |

| Aug 23 | 170 | 1 | 15 | 0.6% |

| Aug 24 | 452 | 97 | 245 | 21.5% |

| Aug 25 | 656 | 1 | 1 | 0.2% |

| Aug 26 | 493 | 7 | 14 | 1.4% |

On Aug 24 this was 10.7% of every LLM turn in the workspace, sustained from 12:00 to 20:00 UTC.

**The part I think is diagnostic**

Every one of the 276 occurrences across all 7 days falls between 12:00 and 20:00 UTC. Not one outside it. That is not just call volume — 00:00 UTC is my busiest hour (864 calls, 1,523 LLM turns) and had zero failures, while 13:00 UTC had 610 turns and 41 failures.

Your own latency telemetry moved at the same time. Median per-call LLM p90 across my calls:

- Aug 20: 640 ms

- Aug 21: 604 ms

- Aug 24: 1,556 ms ← incident

- Aug 26: 1,343 ms

Per-call worst turn went from ~700 ms to 3,014 ms on the 24th.

And the milder `2500ms timeout reached for first token` error tracks it exactly (120-call sample per day, per 100 LLM turns):

| Day | First-token timeouts | Mid-stream timeouts |

|—|—|—|

| Aug 21 | 1.2 | 0 |

| Aug 24 | 9.1 | 10.9 |

| Aug 26 | 5.7 | 0.9 |

- `call_9e9f43e9cd6990cc0e670ced324` — 4 consecutive stalls at 15:25:31, 15:26:13, 15:27:12, 15:27:48 UTC; ended in `inactivity`. Best single example.

- `call_e779bacee9c7cb4b1c87dc385b2` — 3 occurrences, call started 15:16:25 UTC

- `call_94827e48ec393aae092ccc546b4` — 2 occurrences at 15:22:37 and 15:23:12 UTC

- `call_8294be05943d495e76e231f4e70` — 2 occurrences, call started 13:43:09 UTC

- `call_399e9c98d2c159224be4394ccf8` — 1 occurrence at 17:49:17 UTC

- `call_02411e8ea5230754ac2666e12cc` — 1 occurrence at 16:47:29 UTC

- `call_a407c1786d6c92b123c005bed33` — 1 occurrence, call started 13:43:01 UTC

Happy to send the full list of 107 call IDs.

**Questions**

1. Was there a known upstream inference incident on Aug 24, 12:00–20:00 UTC?

2. What is the mid-stream idle timeout on the LLM stream, and is it configurable? The logs suggest roughly 10 s.

3. Is the retry silent by design? A filler phrase during the retry would be far better than 35 s of dead air.

4. Latency is still above baseline on Aug 26 (p50 854 ms vs ~520 ms on Aug 20–21). Is that expected, or is the degradation ongoing? @Shah-Fazal @support4

2

Hello @duncan Thanks for sharing. Checking this with the team, and I’ll get back to you with an update.

3

The time clustering is the most interesting signal here.

Since the failures appear concentrated around 12:00–20:00 UTC and Retell’s latency telemetry increased in the same period, I’d try to separate a platform/upstream-capacity pattern from something specific to individual calls.

For a small sample, I’d compare:

  • 10–20 failed Call IDs
  • successful calls from the same agents and same time window
  • model/provider used
  • whether tools/functions were invoked
  • response length / latency metrics Retell exposes
  • whether the failures cluster around one agent, prompt path, or payload pattern

If failures are distributed across agents/configurations but tightly clustered by time, that would strengthen the case for a platform/upstream issue. If they concentrate around one workflow/tool path, that points the investigation in a different direction.

I’d preserve the failed + matched-successful Call IDs now so Retell can do a clean A/B comparison.

If useful, I’m happy to help structure the redacted comparison set.

4

Hello @duncan Team checked the workspace and the sample calls you sent.

  • Your pattern is real. For your org, we found 478 matching mid-stream timeout log lines across 94 calls during Aug 24 12:00–20:00 UTC. The same signature was visible beyond your workspace: the Aug 24 12:00–20:00 UTC platform-wide query returned 5,577 matching log lines, with multiple orgs represented.
  • These Mid-stream error in stream: Operation timed out entries are not first-token failures. They happen after an LLM stream has already started, then fails to produce the next chunk within the post-start stream idle timeout. The default follow-up-chunk timeout is 7.5 seconds, and the stream path applies that timeout while waiting for follow-up chunks.
  • The first-token hedge/retry is automatic: the first-token path races a first chunk against a timeout and logs ... timeout reached for first token when that first chunk does not arrive in time. After a stream has already started, the mid-stream path does not restart the same response in the same way; it throws the mid-stream timeout instead. That matches why this can feel like silence/dead air rather than a spoken filler.
  • For current status: your org had 28 matching mid-stream timeout log lines across 7 calls on Aug 26 12:00–20:00 UTC, and we found 0 matching mid-stream timeout logs for your org from Aug 27 00:00 through 14:49 UTC. So the specific mid-stream timeout spike is not showing as ongoing in today’s checked logs.

We would treat this primarily as an LLM streaming reliability/latency event rather than a workspace configuration issue. I can’t confirm a named upstream incident from the data checked here, and I would not present prompt/tool trimming as a guaranteed fix for this incident. The direct follow-up is for our product/engineering team to review the mid-stream dead-air behavior and confirm whether a filler/recovery path or customer-facing timeout control should exist for this class of failure.

Thank You