We finally narrowed down what I think the issue is, and implemented a workaround on our end that has resolved the issue for the past few months. However, it is only a workaround, and I believe there’s still a fix to be made on Retell’s end somewhere.
Summary
Requests to api.retellai.com sent over a reused, previously idle HTTP keep-alive connection hang until our client timeout and fail with context deadline exceeded. The request never reaches Retell’s application layer — consistent with your logs showing all requests served very quickly, since the failing ones never arrive.
Disabling connection reuse for Retell traffic eliminated the failures completely. We use the same keep-alive pooling against many other third-party APIs from the same hosts and egress path, and none of them exhibit this behavior.
Client setup
Go net/http with a standard pooled transport (MaxIdleConnsPerHost: 10, IdleConnTimeout: 90s, ForceAttemptHTTP2: true). Client timeouts were 10–60s depending on path; all failed the same way.
What we observed
With net/http/httptrace on failing requests:
| Field |
Value |
reused_conn / was_idle |
true |
idle_time |
large (tens of seconds+) |
| DNS / connect / TLS |
0s (pooled connection) |
| write |
succeeded in <1ms |
| TTFB |
never arrived — hung for full client timeout |
Fresh connections (reused_conn = false) never failed. Failure correlated with how long the pooled connection had been idle.
Likely cause
The connection is torn down or black-holed somewhere on the path without the client being notified, leaving a half-open socket. We write into it; no response ever comes.
A graceful close (TCP FIN or HTTP/2 GOAWAY) or RST would surface in milliseconds and often allow a transparent retry. A hang lasting the entire client timeout means packets are being silently discarded — classic signature of a stateful middlebox (LB, proxy, firewall, NAT) expiring an idle flow and dropping rather than resetting, or a backend terminated without draining.
If you serve HTTP/2, one dead connection stalls every multiplexed request on it.
These failures are invisible in application/access logs: the requests never arrive. Detection requires looking at the connection layer (LB idle timeouts, flow expiry, drain behavior), not request latency.
Our Workaround
DisableKeepAlives = true for Retell only. Confirmed twice: once when first applied, and again when a new code path accidentally reintroduced pooling and the same timeouts returned on only that path until reuse was disabled again.
This adds handshake latency and connection churn on both sides. We would prefer to re-enable pooling.
Asks
- Check for idle-flow expiry (firewall/NAT/LB) that drops rather than resets; ensure LB idle timeout delivers a FIN/RST to the client.
- If serving HTTP/2, confirm GOAWAY is sent/flushed on close; consider server-side PING keepalives.
- Confirm graceful drain on deploy/scale-in.
- Advertise your keep-alive window (
Keep-Alive: timeout=N on HTTP/1.1, and document it) so clients can set IdleConnTimeout safely below it.
- Accept
Idempotency-Key on POST /v2/create-phone-call so clients can safely retry.
Repro: Open a keep-alive connection, idle 60–300s, then POST /v2/create-phone-call on that connection while capturing with tcpdump. We expect request bytes to leave the client, retransmit with no ACK, and no FIN/RST from the server.
Submit your question here: 