We notice our agents have started to skip using tool calls despite saying it will do xyz thing. Here are some calls with this being the case:
call_ba88be5ea2477d2743596d81e8e
call_ee7005d65927c112b756b956419
As far as we know, we havent made any changes to the agent that might result in this kind of behaviour. The most plausible explaination we had was hallucinations due to prompt size. The agent does seem to fetch chunks from the knowledge base on almost every sentence by the caller so I tried removing it and saw mixed results.
Hi Hussain, I’d recommend tightening your prompt and creating conditions for tool usage so that when the conditions are met, the agent will know what to do!
Thanks for the two call IDs — here’s what we found:
Config is clean. Tools (checkAvailability, getAppointment, checkGHLContact, bookAppointment, etc.) are correctly registered on the active LLM version, model is gpt-4.1, and the prompt uses the recommended imperative ‘Run the function X’ style. Other turns on the same two calls fired tools successfully, so this isn’t a registration or version regression.
What’s actually happening. On the Ora Dentistry call the prompt averaged ~13K tokens per LLM request with KB chunks pulled almost every turn. On the Smile Cafe call we can see the failure mid-call: the model narrated ‘please hold while I check availability’ at ~78s, fabricated ‘no openings for Aug 27’ at ~136s, and then finally emitted the actual checkAvailability tool_call at ~165s (which returned 18 slots). That announce → fabricate → belated-or-missing tool_call pattern under heavy context is a known gpt-4.1 adherence miss — the model can produce the narration and a plausible post-tool reply without ever emitting the tool_call token in between.
Most durable fix: move the high-value data-fetch steps to a conversation-flow agent and model them as function nodes. Function nodes fire deterministically on entry, so checkAvailability / getAppointment / checkGHLContact cannot be ‘announced and skipped’. This is the remediation we’d recommend as the durable answer to your question.
In-place mitigations that should reduce recurrence on the current prompt-mode agent (worth trying since your KB-off test was mixed):
Tighten KB retrieval: lower kbConfig.topK from 3 → 1 and raise filterScore from 0.6 → ~0.75 so KB only injects when the match is strong.
We are considering migrating to conversational flow, so hopefully should fix this
However, I did try tightening KB retrieval after making this post myself, and noticed it would still repeatedly retrieve the KB. Bumped up the similarity threshold to 0.95 for one test and it would still do that. Not certain that this is the call with the 0.95 threshold but do believe it was one of the calls I was testing with different KB settings (maybe a threshold of 0.85): call_86b4e7e1a9fa6b9ec9584f3cd4b
Also lowered the chunks down to 2, didnt seem to help.
Sorry but would just like some kind of clarity on why increasing the similarity threshold didnt seem to work and if its something we should actually be employing to reduce the possibility of this happening again while we work on the Conversational Flow agent.
Its happened a few more times and ive increased the threshold and reduced the chunk count for each agent but would rather not be making changes in vain.
Happened to come across another call, where while the similarity threshold was set to the default 0.65, it still retrieved the KB on even sentences like “Yes” “I’ve been there before”, which imo do not require a retrieval under any circumstances.
call_fc83cc9c433ff79a9917a188e97
We assumed that KB retrieval is something the agent should be explicitly instructed to do, but doesnt seem like this is the case here. Not sure if something changed but once again not really an issue we’ve faced before.
Hello @hussain Why raising the similarity threshold / lowering topK didn’t stop retrieval from firing: filterScore and topK don’t gate whether KB retrieval runs — they only bound what comes back. The retrieval event fires whenever the model decides to consult the KB, and only after that does the filter/topK cap apply:
topK = max chunks returned per invocation (0.95 threshold → often 0 chunks survive, but the retrieval call still executed).
filterScore = minimum similarity for a chunk to survive. At 0.95 you’re essentially forcing zero-chunk returns — the KB stops answering but does not stop being consulted. That’s why your timing/latency views still show KB events per turn.
What actually drives the invocation frequency on this agent:
knowledgeBaseIds is attached at the LLM (flow) level, so retrieval is model-decided on every LLM turn.
Your prompt has explicit # Knowledge Base and # Insurance sections telling the model to consult {{knowledge_base_name}} / {{insurance_list}} for anything outside the scripted flow — that steers the model to trigger retrieval on almost any user turn that isn’t a pure booking-step reply.
So 0.95 threshold + 2 chunks didn’t reduce the number of invocations, it just made every invocation return near-nothing. The knowledgeBase.num counter you see on the call record is the count of invocations, not chunks delivered.
Should you keep using threshold/topK as a mitigation for the tool-skipping issue?
Short answer: no The tool-skipping we diagnosed earlier is an adherence problem under heavy context (announce → fabricate → missed tool_call emission), not a chunk-count problem. Raising filterScore to 0.95 doesn’t reduce prompt weight much (the KB section is small if it’s returning 0 chunks) and it does break KB answering. Two better levers while you migrate to Conversation Flow:
Cut retrieval frequency at the source: tighten the prompt language around when the KB should be consulted (e.g. explicit “only consult the KB if the user asks a services/insurance/general question outside this flow”) — the model listens to that far more than to filterScore.
Or drop knowledgeBaseIds from the LLM for the duration of testing and inline the small amount of insurance/hours info directly in the prompt as static text. That eliminates the retrieval turn entirely.
For the tool-skipping specifically, the migration to Conversation Flow is the real fix — function nodes hard-gate tool execution instead of relying on the model choosing to emit a tool_call token.