Agent is not Switching Language even on consumer's request

Facing issue:
Agent is not switching language even on consumer’s request.

Details:

  • I have added all node prompts, examples, global prompt in English.
  • I have selected MultiLingual
  • I also have kept guardrails in Global Prompt as follows:

## LANGUAGE & MIRRORING

  • strictly detect_language only for: Hindi, Hinglish, Marathi, Gujarati, Tamil, Telugu, Kannada, Odia, Malayalam, Bengali, Urdu, English (India).
  • Guest mentions any of the above language, switch to the mentioned language immediately
  • after switching-language continue in same language, unless guest requests otherwise.
  • Mirror caller language immediately.
  • Match formality, pace, and emotional energy.

Emotional & Energy Alignment:

  • Excited → warm and enthusiastic tone.
  • Calm → composed and clear.
  • Hesitant → reassuring and slower.
  • Urgent → concise and efficient.

I asked the same question to ASK AI, below is the response I got:

Based on the information sources, language switching in Conversation Flow agents requires specific handling because language switching only happens for the current node—when the agent transitions to another node, it reverts back to English.

Solution for Conversation Flow Agents:

To maintain language consistency across nodes, you need to:

  1. Extract the user’s language preference into a dynamic variable (e.g., {{user_language}}) after the first node where language selection occurs.

  2. Add language instructions to every subsequent node. Here’s the recommended prompt structure to add at the top of each node:

Always speak in {{user_language}} throughout this node.  

Assume the user only understands {{user_language}}.

Only listen for and recognize speech in {{user_language}}.

Transcribe and interpret the caller's response exclusively in {{user_language}}.

Do not use English unless {{user_language}} is explicitly set to "English".

  1. Update your global prompt to include:
**Speak only in the user's language ({{user_language}}).**

Alternative Approach:

If maintaining language-specific prompts in every node becomes too complex, you can use agent swap to swap to an agent configured for the specific language. However, this would require maintaining separate agent copies for each language.

Important Note:

When you select the language of the agent in global settings, you’re setting what language the agent will understand (the language of user audio). To make the agent respond in a specific language, you need to write instructions in the prompt such as “respond in Spanish”.

For additional support with your specific configuration, check out the documentation or ask the community for help.

NOT SATISFIED with the AI Response:

  • Both the recommended approach, increase TOKEN COUNT Drastically and are not practical.
  • I have 40+ NODEs handling various scenarios.
  • What is the role of Global Prompt, if we cannot control/manage NODEs centrally?
  • Why is 2nd approach recommended, when we have MultiLingual as an option?

Hey @info29

Could you please share the call IDs and the Agent ID? where you faced that issue. This will help our team investigate the issue more effectively.

Thank you!

Agent ID: agent_676c44a96e38f971e9c40accc7
Call Session ID: call_8a59799c634a7ee403f4b5707a1

I am also facing a lot of issues (so many calls) where agent is unable to capture the words (indian accent) correctly.
Error (inaudible speech).
Call ID: call_21ed285751185d987e685ab4a9a
Guest was asking for Nicco Super Bowl, agent interpreted Niku super board.

Hey @info29

I’ve escalated this to our team for further review.

We’ll update you as soon as we hear back.

Best regards

Hey @info29

We’ve noticed that the sound quality seems to be poor. Could you also confirm if your customers are primarily based in India? If so, we recommend using an Indian language instead of the multilingual mode, as it may improve overall performance and audio quality.

Thank You

The sound quality is not poor. Each Marathi word is very clear.
Call Session ID: call_8a59799c634a7ee403f4b5707a1

Agent infact replied that it knows marathi (replied in Marathi language only), however only that statement was in Marathi, rest agent continued talking in English again.

Our customers are from all over India.
All of these are Indian Languages - Hindi, Marathi, Gujarati, Tamil, Telugu, Kannada, Odia, Malayalam, Bengali, English (India) - from different states of India.
Also earlier all of the above languages were visible in the dropdown - now Telugu, Odia, Malayalam, Bengali is not visible. (is this dependent on the ElevenLabs Agent selected?)

When our voice-agent picks-up a call, user can speak any of the above languages. so sticking to only a single language isn’t an option.

Moreover, since I have initiated this thread, I noticed Multilingual option has an :warning: error sign (warring or alert symbol)

Hey @info29

Multilingual recently received an update allowing you to select only the desired languages, which should help resolve this issue. This change is also why you see a warning or alert symbol.

Thank you for the update.
I have chosen the relevant languages and published the agent. I will update this thread, if the issue persists.

Please if you can also check and confirm, why the following languages aren’t visible now:

  • Bengali, Gujarati, Malayalam, Telugu

@mark1
Call ID: call_a6e5080decdc3509bc67f7d06fa

Though I have selected Multilingual & only indian languages:

  1. Agent spoke DATE in perhaps russian, non-Indian, non-selected language (listen at the last statement of the agent). Error occurs due to transition to Library component, I don’t see if Library component has same language configurations as Main, I don’t see option to configure the same.
  2. while talking in regional language, agent is capturing variables in regional language, how to configure to capture details in English only.?
  3. Still Agent is making a lot of errors in capturing correct words, eg: for Play N Learn → agent captured Plane Learn,

Hey @info29

Team checked this and

  1. It appears to be a temporary issue with ElevenLabs causing confusion about which language to use, and the selected language will affect the transcription, not the TTS

  2. Prompt the agent to extract variables in English.

  3. Add these keywords to the boosted keywords list.

Thank You

Boosted KWs already had Play N Learn:
More such cases:

  • Splash captured as supplash.
  • crescent captured as christian
  • WetnJoy captured as wetngo

@mark1
Please help resolve this on priority, it is our peak season and it is directly impacting our revenue. Our current Retell billing is not resulting into ROI.

Refer: call_1f7605654486d00537df931a395
Guest is talking in Hindi.
Transcript also shows Hindi language, however agent continues in English. Similar across many calls.
I had already shared the prompt earlier in the thread.

Refer: call_8b25f5edd6a2fd86234af636f22
Guest is talking in Marathi.
Transcript also shows Marathi language, however agent continues in English. In this same call, Agent captured Boosted KW ‘Wetnjoy’ as ‘Wait and Joy’.

Also the earlier issue doesn’t seem to be at ElevenLabs, the date is spoken in different language everytime. This is primarily due to transition to Library Component (which doesn’t have language configuration). Otherwise agent would be talking in non-Indian language in Main component as well.

Hey @info29

I’ve followed up with our team again to take another look at this issue.

We’ll keep you updated as soon as we have more information.

Best regards

Hey @info29

You need to prompt the LLM to respond only in the desired language. Selecting the language in settings affects transcription but not the LLM’s spoken output.

Also, prompt the LLM to speak dates as text rather than numbers.

:warning: :police_car_light: READERS ALERT :police_car_light: :warning: **
**
REPLIES NOT HELPFUL:
I don’t see efforts being taken by the RETELL team to actually look for what issues their consumers are facing, instead the focus is only on replying back with some random, vague instructions. I am highlighting and repeating same errors with multiple ref. CALL IDs in the entire Thread, I don’t think anyone is even going through them, because if they did, they would understand the issue. Either the team have gone through and are unable to resolve and admit it publicly, or the team haven’t gone through it.

:enraged_face: Do NOT mark any thread as SOLVED unless I consumers confirm. This is so UNETHICAL and MISGUIDING to readers. :enraged_face:

@mark1

I have burnt almost $150+ in just testing. This is really unhelpful.

GLOBAL Prompt (which covers instructions and agent is not following them at all) covers:
LANGUAGE & MIRRORING

  • Guest mentions any of the Hindi, Marathi, Gujarati, Tamil, Telugu, Kannada, Odia, Malayalam, Bengali languages, switch to the mentioned language immediately.
  • If the user switches language, switch immediately.

Agent ID: agent_676c44a96e38f971e9c40accc7
Call ID: call_857974e0e39581b2dcd92cb4758
Guest is talking in English and Agent identifies it as Tamil.
Inspite of giving instructions to capture variables only in English, it captured raw_venue_name in Tamil.

Hello @info29

Sorry for the inconvenience. I’ll forward this to our team for further investigation.

Best regards