BUG: Outbound Agent Not Processing Initial User Response

Hi Retell Support,

We are seeing a recurring bug/issue with one of our outbound agents:

The user answers the call and clearly says something like “hello”, “yes”, or “it is”, which is captured in both the audio and the transcript.

However, the agent repeatedly says the opening line:

“Hi there, is this [user full name]?”

This problem is happening even after the user has already responded or confirmed their identity multiple times.

This client has been using Retell for outbound calls for around 18 months, and we have used the same “user speaks first” style setup from day one.

We have never seen this specific behaviour before, but it has begun to appear repeatedly over the past few weeks.

Our prompt includes the opening instruction:

“Immediately after the user speaks when answering the call, output this EXACTLY: ‘Hi there, is this {{full_name}}?’”

The problem is that the user’s answer appears to be detected in the transcript, but the agent behaves as if it has not heard or processed the response, and repeats the same opening confirmation question multiple times.

This is creating a poor caller experience because the caller is clearly responding, but the agent keeps talking over them or re-asking the same question.

Affected agent:
agent_6469ced6df072b5c1788a25e60

Org ID:
org_vKVl7EVrGjqQoEMi

Affected call examples:

  • call_abd31ebafc39ef852120bb9cbcf
  • call_62e04442770e1e9ace35cdfebdf
  • call_2519f5e4d932609ad29b6dbea7c
  • call_aef5b86e20c2f7430f5a44e3512
  • call_4ed26d590dc202b52cd658e79db

Examples of the issue:

Example 1 - call_abd31ebafc39ef852120bb9cbcf
The user says “Hello?”, then “It is. Yeah.”, then “It is. Yes.”
Despite this, the agent repeats “Hi there, is this Stefan Fletcher?” multiple times before eventually continuing.

Example 2 - call_62e04442770e1e9ace35cdfebdf
The user says “Mhmm. Hello?”, then “Hello?” several times.
The agent repeatedly asks “Hi there, is this Martin Ellis?” and does not appear to progress.

Example 3 - call_2519f5e4d932609ad29b6dbea7c
The user says “Hello?”, then “It is.”, then “Yes. It is.”, then “Yes. Yes.”
The agent repeats “Hi there, is this David Rodney Smith?” several times before eventually continuing.

Example 4 - call_aef5b86e20c2f7430f5a44e3512
The user says “That was quick. Hello?”, later says “Hello?”, and then says “Your phone broke.”
The agent continues asking if this is John Williams and appears not to process the caller’s responses correctly.

Example 5 - call_4ed26d590dc202b52cd658e79db
The user says “Hello?”, then “Yes.”, then “This is Frankie Davis.”, then “Yes. It is.”
The agent repeats “Hi there, is this Frankie Davey?” multiple times before eventually continuing.

What we need help checking:
Can you please review these calls and confirm whether there has been any recent change or regression affecting outbound calls using user-speaks-first behaviour, start-of-call turn-taking, ASR/VAD detection, interruption handling, or the way the agent processes the first few user utterances?

Specifically, can you check:

  1. Whether the user’s speech was correctly detected by ASR/VAD at the start of these calls.

  2. Whether the agent received and processed those user turns before repeating the opening message.

  3. Whether interruption/barge-in behaviour is working correctly on this agent.

  4. Whether there is any issue with audio/transcript alignment at the beginning of the call.

  5. Whether any recent Retell platform-side changes could explain why this has started happening in the past few weeks, despite this client using Retell outbound calls for around 18 months without seeing this behaviour before.

  6. Whether there are any recommended agent setting changes to prevent the agent from repeating the opening confirmation question after the user has already answered?

    Thank you.

Hello @Sean_Whelan_AYB_AI Thanks for the details, I have escalated this to the team for a review. Thank You

Thank you @mark1 , appreciate it

Hello @Sean_Whelan_AYB_AI we reproduced this. It’s fixable on your side, not an ASR or platform issue: the caller’s speech reached the agent every time, and the greeting played fully (no interruption).

The cause is Task #1 in your prompt: “Immediately after the user speaks… output this EXACTLY: ‘Hi there, is this {{full_name}}?’” It has no “only once” rule, so every “Hello?” makes the agent re-say the greeting. Two changes fix it (we tested them):

  1. Scope the greeting to the first turn. Replace Task #1 with: “ONLY on your very first turn, output this EXACTLY: ‘Hi there, is this {{full_name}}?’. On every later turn, never repeat it — respond to what the caller said and continue.”
  2. Set responsiveness back to ~0.95. Raising it to 1.0 on June 8 is what started splitting short replies into multiple turns and re-triggering the greeting.

Publish a new version with both and send us a fresh call to confirm.

Thank You

Thank you @mark1 - we have deployed those fixes and will review the calls manually for the next week or so to ensure it’s working correctly

@Shah-Fazal @mark1 - Hi guys, we implemented the changes you suggested last week.

However, the issue is still happening, as seen in call ID:

call_4aa790e5df718c4c5d961adaff8


Also, as a separate issue with the same voice agent + org ID mentioned above, can you please investigate why this call failed to transfer?

call_3d1060f0314e28d3cf68e817d5c

Thank you

Hey @Sean_Whelan_AYB_AI I have escalated this to the team for a review.

Hey @Sean_Whelan_AYB_AI Team will look for another fix, in their tests it did always work.

Would you be able to use the AI speaks first and have the intro line there instead?

Thanks

Thanks @mark1, we can’t use AI speaks first because this agent is for outbound calls.

In our testing with AI speaks first on outbound calls, in most cases, the AI speaks before the user actually answers the outbound calls, and performance decreases due to this.

@Sean_Whelan_AYB_AI Noted we will see if there’s something else we can change to make this work.

Thanks Mark, it had been working fine for ~18 months so surely there must be some fix

@mark1 This seems to be fixed now (so far anyway) - marking as solved in the meantime - I’ll message here if we see it again - thank you for your help!