Post-chat analysis started to hallucinate without me changing anything

Support Help
1

Background

We have a variable we extract from every chat called __preferred_modality which should either be “voice”, “sms”, or none. It should only be set when the customer explicitly requests another modality and should always be none if there is no reply or nothing mentioned.

Our prompt for this typically works completely fine (using gpt-4.1). We can measure this in our data by seeing the % rate of __preferred_modality being set when there was no user reply or no modality mentioned within any of the user replies.

Previous Issues

For several weeks we were at 0% but then something happened (presumably on Retell’s end) without us changing the analysis where this shot up to upwards of a 90% error rate between August 11th and August 25th before dropping back down to 0-1% again without us changing anything.

Current Issue

This error rate had been at 0% since then until a couple days ago on September 15th where it has since shot back up to 40%.

Here are some examples of what I’m talking about:

Good Examples:

  • chat_550d05f4b6ccf0f0a2524bdf77d (no user reply so __preferred_modality not set)

Bad Examples:

  • chat_a7af5611afa42675ab27684f705 (no user reply but __preferred_modality was set anyways)
  • chat_f5d4e01046d20be2fb401dea2c8 (user only replied “Hey Ok” but __preferred_modality was set anyways)
  • chat_5f96c5469c69b0a621b7a5300c7
  • chat_c5638e2389074da254abe9f133f

In many of these cases it seems to be getting set based on the Agent itself saying “Heads up {first_name}, Ryan from {company_name} will be calling in a few minutes. Talk soon!”.

The prompt we’re using is incredibly clear and given that this model has been 100% successful in the past and changes performance drastically out of nowhere, I have to assume it is some change in behavior with how Retell is providing the analysis prompt to the model:

Enum: which channel does the human prefer for ongoing engagement?
Allowed values: `sms` or `voice`. Leave unset if no explicit channel preference was
stated. Never set to `null` or `None` as a string -- omit the variable entirely instead.

Only set when the human explicitly names a channel. Do not infer a preference from context alone (e.g., the fact that they are texting now does NOT mean they prefer SMS). Ambiguous acknowledgements like "yeah" or "ok" must NOT set this variable.

`sms` examples:
- "Please text me" -> sms
- "Just message me instead" -> sms
- "Text me later" -> sms

`voice` examples:
- "Just call me" -> voice
- "I prefer phone calls" -> voice
- "Call me, don't text" -> voice

Do NOT set:
- "Yeah" / "Sure" / "Ok" -> no channel mentioned
- "I'm busy" -> no channel mentioned

I’ve run some additional experiments where I’ve added the following to the prompt as well, but there is still around at 30% error rate.

GROUNDING: Set this from the HUMAN's own words only. If the transcript has no human turn -- only agent/system messages (for SMS, the pre-call SMS "Heads up NAME, AGENT from PARTNER will be calling in a few minutes. Talk soon!") -- set `false`. That pre-call SMS is the agent's own announcement of the upcoming call; it is NEVER a lead request to reschedule. Never infer a defer from an agent or system line.

I have also tried changing the model to various other versions of gpt-4.1, gpt-5, claude-4.5, gemini-3.5-flash-lite and they all have issues with this as of a couple days ago.

Questions

  1. Given that we have not changed anything about our agent config, can you confirm if there were changes on Retell’s side that would have caused this?
  2. What do you suggest I change to improve the performance of this analysis variable back to before a couple days ago when it was working fine?
  3. I see this Retell Platform Changelog. Is there a more detailed technical changelog I can monitor to keep an eye out for potential breaking changes in the future? With this I would be able to provide more details on future issue reports.

Thank you in advance, your help with this issue is appreciated.

2

Hello @sperales Thanks for sharing the details. We’ll use this information to investigate the issue further.

Thank You

3

Hey @sperales Looking at your chat agent config (agent_08136ae46bc82fae768d0dc472) and the analysis pipeline, the root cause is schema-side rather than prompt-side:

  • Each item in post_chat_analysis_data supports a required flag. When it is anything other than false, we add that field to the JSON-schema required array sent to the model and we do not strip empty/null values from the model’s response. Your __preferred_modality are currently configured without required: false, so the model is being told it must return a value — no matter how strong the “leave unset” wording is in the description. When the underlying analysis model becomes more strict about honoring required, it starts inventing a value from the only text available (the agent’s pre-call SMS), which matches exactly what you’re seeing. That also explains why swapping to other models did not help.

  • Please set “Allow empty value” on __preferred_modality and any other custom field that should be omittable (e.g. __reschedule_at, __reschedule_at_freetext, __new_number, __unanswered_questions, __languages_detected, accidental_signup, plan_selected, etc.). With “Allow empty value” on:

  • the field is omitted from the JSON-schema required list, so the model is allowed to skip it

And i will route this to engineering team to answer your other questions.

Thank You

4

Hello @sperales Team found a fix for this, basically, for PCA fields, don’t use a string field for everything. Using strings for all the fields is causing some issues, so go through the fields, and make the true/false ones Booleans, and the ones that should only output a range of values Enums. Adding more rules to make the model only return the correct string values is not as effective as using the correct type in the first place. For example, I have found chats where the preferred_modality is “false”, which is because using prompts to limit the values didn’t work in this case, and previous string values can affect later PCA values. With this change, you should be able to remove a lot of rules in the PCA field descriptions about the valid values, which should also improve performance.

Additionally, it’s probably a good idea to also keep the order of the defined PCA fields the same when making changes to the agent. This is because as mentioned above, previous PCA values can affect future PCA values, so keeping the order the same reduces variabliity across agent versions.

In summary:

  1. Change the fields to the appropriate types
  2. Reduce the description rules per field
  3. Keep the order of PCA fields stable

Thank You