Agent/caller speech mis-ordered in transcripts — caller answers after the agent finishes but is recorded as interrupting mid-sentence; breaks conversation-flow transitions (Japanese / Fish Audio TTS)

Support Help
1

Summary

On an intermittent subset of our calls, the transcript orders agent and caller speech differently from what actually happened on the call. The caller waits for the agent’s question to finish and then answers; the transcript records them as interrupting mid-sentence, splitting the agent’s utterance and placing the answer inside the question.

On every one of these calls, the conversation flow then goes wrong at exactly that turn: the transition edge that should fire on the caller’s confirmation doesn’t, the node LLM improvises a final-sounding wrap-up in-node, and the caller hangs up before the terminal node that speaks our client’s closing message (business hours / callback promise). Multiple hotel clients have reported the missing message. Calls where the transcript ordering is correct transition cleanly every time.

What the caller heard vs what the transcript recorded (call_606f506ff90c580462902f2a37e, phone call)

From the multichannel recording: the agent channel’s audio for the confirmation question ends at 72.45s; the caller’s「はい」is at 73.8s — 1.35 seconds after the agent finished. The real exchange:

▎ Agent: ありがとうございます。折り返しのご連絡は、お電話いただいている「ゼロ ゴー ゼロの…」の番号でよろしいでしょうか？
▎ Caller: はい

The transcript records:

▎ Agent: ありがとうございます。折り返しのご連絡は、お電話いた—
▎ Caller (interruption): はい
▎ Agent: —だいている「ゼロ ゴー ゼロの…」の番号でよろしいでしょうか？

As written, the caller confirms a number they haven’t yet heard. The meaning of the turn is changed, and the flow behaves accordingly.

Timestamp findings

We compared transcript_object word timestamps against the multichannel recordings for six calls:

  1. On the mis-ordered calls, every agent-side character has an identical duration of ~0.393s, punctuation included: あ(66.59–66.99) り(66.99–67.38) が(67.38–67.77)… (call_a718…, response 15). On the correct calls — same agent, same voice, same half-hour — character durations vary naturally (0.08–0.2s, gaps at punctuation).
  2. As a result, agent utterances on the mis-ordered calls are 1.3–1.5× longer in the transcript than in the audio. In call_606f… the final utterance’s word timestamps extend to 129.8s while the recording ends at 123.5s — 6.3s of transcript past the end of the audio file.
  3. Caller-side timestamps match the caller audio channel within ~0.1s on all six calls. Only the agent side is inflated.

So the mis-ordering isn’t random: accurate caller timestamps are being merged against an agent timeline that runs long, and the caller’s answer lands inside the phantom tail of the agent’s utterance.

Environment / calls

  • Agent voiceai_stg_3d5a9af7 / agent_041175132263898b633743a68f (v0); conversation flow conversation_flow_6be81a0f847b, shared component conversation_flow_component_2b28901e3c47; Japanese, fish_audio_tts. Calls 2026-07-23 03:53–04:19 UTC.
  • Mis-ordered (~1.4× stretch): call_a718f3d0728598fd257d5b35de3, call_605830e9f5fae1fbc2a5626e429, call_606f506ff90c580462902f2a37e (phone), call_1f02569b347d42de3007cf4f83a
  • Correct (~1.0×, same agent/voice/window): call_3df6c4fb9f3a4e24de23ba3b44c, call_85238edd84bfa1b4639c35320b3

Questions

  1. What causes agent word timestamps to come out as uniform ~0.39s/character on some calls and real alignment on others? Is this a known fallback path for Fish Audio / Japanese voices?
  2. Does interruption detection and edge evaluation use this same timeline? That would match everything we observe.
  3. Is there a fix or configuration workaround you recommend?

Channel-split audio analysis (per-channel speech segments vs word timestamps, all six calls) available on request.

2

Hello @andrew3 Thanks for the detailed write-up and for sharing the Call IDs. I’m checking this with the team and will get back to you as soon as we have an update.

Thank You

3

We’re still seeing this. Here are some new examples, both from 2026-07-28 (~01:46–01:48 UTC), same conversation flow, Japanese Fish Audio custom voice (custom_voice_c9cd94fd5a53bca14cc385c189, model s2-pro).

call_551f12e0f3b13c606741f76ff70

The recording (multichannel WAV, per-channel speech segmentation):

  • AI finishes the confirmation question「チェックイン前にお荷物をお預かりできるかどうかのお問い合わせですね。よろしいでしょうか？」at t=20.65s
  • Caller answers「はい、お願いします」at t=21.3–22.35s — 0.65s after the AI finished. Clean turn-taking, no overlap.

The transcript instead claims:

  • AI turn: t=16.46 → 22.56s — about 1.9s longer than the actual audio
  • Caller turn: t=21.27 → 22.15s — placed inside the AI turn, ordered before the AI’s final chunk

So the flow’s edge classifier sees: AI asks「…よろしいでしょうか？」as the last event, with no caller answer after it → takes the else edge → re-asks a question the caller already answered. Node log confirms Confirm Topic → Re-ask Topic.

call_c4df4480a901a3695bdfebb44b0

Same pattern, worse transcript:

  • Recording: AI finishes the question at t=18.9s; caller answers「はい、お願いします」at t=19.55–21.2s (again ~0.65s after, no overlap)
  • Transcript: AI turn t=14.9 → 21.0s (~2.2s inflated), and the caller’s answer is split into two fragments —「はいお願いしま」ordered before the AI’s final chunk, and a stranded「す」after it. The classifier evaluates「す」as the caller’s answer → Confirm Topic → Re-ask Topic (public.log 01:47:56.948).

The pattern across both calls:

  • AI turn start timestamps match the audio; end timestamps are inflated ~2s — the TTS duration appears overestimated
  • Caller timestamps match the audio, so the inflated AI turn swallows correctly-timed caller responses and renders them as mid-sentence interruptions
  • Our QA reports this reproduces with Hoshi-derived Fish Audio voices but not with Mana, so the duration error may be voice-specific

This is not cosmetic — node transition classification consumes the misordered transcript, so flows re-ask confirmed questions (and previously, broke our callback flow the same way). Could you check how agent-turn end times are computed for Fish Audio voices (estimated vs. actual playback), and whether edge classification can use actual utterance completion order?

4

Update — a third call, Cartesia voice, showing a different timing failure:

call_72a75c2c33ced2245d82867cef3 (Cartesia voice).

In this call the agent-turn timestamps match the audio (e.g. transcript claims 31.18→36.30; the recording shows 31.2→36.25). There is no turn-duration inflation like in the two Fish Audio calls above.

What we observe:

  • In the multichannel recording, the agent finishes the confirmation question at t=19.15, and the caller answers「はい」at t=20.05–20.25 — 0.9s after the question, no overlap.
  • The transcript contains no user turn at that point. Its first user turn is「はい」timestamped t=35.53–35.81 — 15.5 seconds after the caller audibly spoke.
  • The caller channel contains exactly two speech events (t=20.05 and t=37.25; remaining activity on that channel cross-correlates with the agent channel, i.e. it is echo of the agent’s audio). The transcript contains exactly two user turns:「はい」at t=35.53 and「はい」at t=37.25.
  • With no user turn registered after the first question, the agent re-prompted at t=31.2. The flow transitioned only after the caller’s second はい (t=37.25); node log shows Confirm Topic → Route Confirmed Topic at 02:10:35.828 UTC.

So the caller’s first confirmation appears in the transcript 15.5 seconds after it was spoken, and the conversation flow re-asked a question that had already been answered.

Across the three calls: on the Fish Audio calls, agent-turn end timestamps are ~2s later than the audio and the caller’s on-time answer is rendered as a mid-sentence interruption; on this Cartesia call, agent timestamps are accurate but the caller’s short「はい」appears in the transcript 15.5s after it was spoken. In all three, flow transitions acted on the transcript picture and re-asked confirmed questions.

5

@andrew3 I’m checking this with the team and will get back to you as soon as I have an update.

Thank You

6

New example from 2026-07-28 08:27 UTC, this one with an audio defect as well as the timestamp one: a sentence was spoken twice, a clause of the answer was not spoken at all, and the transcript shows the text once.

call_00bca87a1cdb72fb8ca8f5d4567 (agent voiceai_stg_0c5a7e08_ja-JP, Fish Audio custom voice custom_voice_7f62bf65cdc2ffdf72eaf473c9, ja-JP)

The LLM generated the answer text once — the node log shows a single knowledge-lookup and a single answer node, no re-entry. The turn text, as three sentences:

  • S1:「ダイ・チュウ・ショウ合わせて20の宴会場がございます。」
  • S2:「大・中宴会場、会議向けの小宴会場、和室宴会場、高層階のスカイバンケットなど種類も豊富です。」
  • S3:「最大収容人数は、サンガイの大宴会場「鳳」をビュッフェ形式でご利用の場合、ゴヒャクメイ様となっております。」

What the agent channel of the multichannel recording contains:

  • 41.3–45.7s: S1
  • 45.5–50.5s: S1 again
  • 50.9–56.6s: the second half of S2 —「和室宴会場、高層階のスカイバンケットなど種類も豊富です。」. The first clause,「大・中宴会場、会議向けの小宴会場、」, is not spoken
  • 57.2–65.6s: S3
  • 65.8s: agent audio ends; caller hangs up at 69.1s

So the audio for S1 was delivered twice, and the first clause of S2 was not delivered at all.

The transcript for this turn contains the text once, in full, including the clause that was never spoken, on the same synthetic timeline as the earlier calls: every character uniformly ~0.42s, the turn ending at 71.4s — past the end of agent speech (65.8s) and past the end of the recording (69.1s) — with the tail words at zero duration.

This is affecting a lot of our production calls currently so any updates or guidance are appreciated.

Thanks,
Andrew

7

Hello @andrew3

This is something related to the provider.

For short term fix can you set Cartesia custom_voice_40e9b59b6a413160becb57cd80 as your main voice?

It will mitigate the issues.

Thank You

8

Thanks! Can you share more details? Is Fish Audio likely to get better and is there anything that Retell can do on their side to improve things? Fish Audio voices provide the most consistently accurate and natural-sounding Japanese speech.

Andrew

9

Hello @andrew3 We are actively investigating the solution right now, Its because our transcript estimation is not designed for japanness character and causing the issue.
Estimated fix will be launch around Tuesday when you can use Fish Again.

Thank You

10

Thanks for the update! Looking forward to this being fixed.

Andrew

11

Hi,

Have you been able to push a fix yet? Let me know if you have any updates.

Thanks,

Andrew