Hey everyone — I’ve been building voice agents on Retell for a few months now and keep running into the same frustrating thing.
When the same person calls back a second or third time, my agent has zero memory of the previous conversation. Treats them like a complete stranger every time.
Curious how others are dealing with this:
Are you using any workaround?
Is this killing your client relationships?
How much time do you spend rebuilding context manually?
Not looking to sell anything — genuinely trying to understand if this is just me or if everyone is dealing with it. Drop your experience below
You need to set up a separate database (I suggest using Supabase SQL) and a separate Backend for this (I used N8N, but make.com can work too). Retell doesn’t have this function currently, so you need to build it yourself or ask an expert for this.
As mentioned above by admin19, currently you would need to set it up manually, but it would be nice if Retell introduces a feature like this in the future.
Yeah, you’re not imagining it. Retell doesn’t hold memory between calls out of the box. Every call starts cold, so the agent treats a repeat caller like a stranger. Most people building on the platform hit this same wall.
The fix is to build the memory yourself and feed it back in per call. It’s less work than it sounds. Set up an Inbound Call Webhook on your number. Before the call connects, Retell sends you the from_number. You take that, look the caller up in your own store (I use n8n with a database behind it, but a CRM or Airtable works fine), and return their history as dynamic_variables in the webhook response. Retell drops those straight into the prompt and begin message, so the agent can open with something like “good to hear from you again” and reference the last chat.
Then after the call, use post-call analysis or the call webhook to grab a summary and save it back against that same phone number. Now it’s waiting there for next time. So the platform is stateless by default, but the phone number becomes your key and you bolt a memory layer on the side.
On your time question, once the plumbing is in you’re not rebuilding context manually at all, it’s automatic. The manual pain only shows up if you’re running without a webhook and a store behind it.
How are you handling your call data at the moment? Tell me what you’ve got for storage and I can point you at a cleaner setup.
You are definitely not alone in this! Since Retell agents handle each call as an isolated session, they don’t natively remember returning callers out of the box. To solve this in our own setups, we ended up building a custom workaround on our end using Supabase as our database and writing a dedicated API to manage the memory layer. Whenever a call comes in, our system intercepts it using the caller’s phone number as a unique identifier, queries the database for past interaction history or notes, and dynamically injects that context into the agent’s prompt right before the call initializes.
It completely solves the issue of treating clients like strangers, but it does require some engineering effort to get right. The biggest factor we ran into is optimization if your database lookup or API response takes too long, it introduces a frustrating delay right at the start of the call. Keeping the data structured cleanly and keeping the query fast is essential. If you don’t want to maintain a full database backend yourself, you can achieve a similar result by using n8n or Make.com as a middleman to pull past data from a CRM like GoHighLevel or HubSpot on the fly, but building a custom API wrapper around your own DB is definitely the most robust way to handle it.
