Performance-related configurations and noticeable latency during live calls

Support Help
1

Hello RetellAI Support Team,

We at Galaxy Weblinks are currently exploring RetellAI to develop a voice agent for our organization. We have successfully set up an assistant with a configured knowledge base, and it is working well in terms of providing accurate responses to user queries.

However, we are experiencing noticeable latency during live calls, particularly the delay between the user’s input and the assistant’s response. We attempted to reduce this by updating the assistant’s system prompt to include filler responses while processing, but the delay still persists.

Could you please advise if there are any configurations, best practices, or optimization techniques available in RetellAI to help minimize or eliminate this latency? We would appreciate guidance on:

  • Recommended audio or model settings

  • Any performance-related configurations

  • Best practices for reducing response time in real-time voice interactions

Please let us know if you require any additional details from our side. We would also be happy to connect for a discussion if needed.

Your guidance would be greatly appreciated.

Thank you and regards,
Preetam Pawar
Galaxy Weblinks Pvt. Ltd.
+91-7000293014

2

Based on the available sources, here are specific recommendations to reduce latency in your Retell AI voice agent:

Model and TTS Optimization

Change to a Faster LLM: Use a more efficient language model to reduce response time and improve performance.(1)

Change to a Faster TTS: Implement a faster text-to-speech system for quicker audio output and smoother conversations.(1)

Retell AI typically achieves sub-600ms to 620ms average end-to-end latency with optimized configurations.(2)(3)

Turn-Taking Model Configuration

Retell AI’s proprietary turn-taking model uses predictive timing to ensure smooth transitions between speakers.(4) The model predicts when to take turns, ensuring smooth transitions and preventing awkward pauses.(4)

Better Turn-Taking Model: Implement a more sophisticated turn-taking model to accurately detect the end of a user’s turn, preventing the AI from interrupting or prematurely responding.(1)

Performance Targets

Production voice AI agents typically aim for 800ms or lower latency to maintain conversational flow.(2) Retell AI demonstrates exceptional performance with:

  • Time-to-First-Token: 180ms average(2)
  • End-to-End Latency: 620ms average(2)
  • Barge-in Response: 140ms average(2)

Knowledge Base Considerations

From Wednesday, February 04, 2026, I note that large knowledge bases in prompts can impact performance.(5) One user reported that including a 3202 byte markdown file directly in the base prompt as a workaround may have contributed to slower response times.(5)

Additional Optimization Techniques

Retell AI’s streaming architecture minimizes latency through:

  • Edge deployment: Distributed processing reduces geographic latency(2)
  • Streaming optimization: Chunked audio processing minimizes buffering delays(2)
  • Predictive pre-loading: Context anticipation reduces response preparation time(2)
  • Adaptive bitrate: Dynamic quality adjustment maintains performance under network stress(2)

Troubleshooting Responsiveness vs Interruptions

If you’re experiencing issues balancing low latency with preventing interruptions, the turn-taking model’s adaptive learning can help.(4) The model learns from interactions and adapts to different conversational styles.(4)

For specific configuration assistance with your setup, I recommend:

Would you like me to provide a general answer based on broader voice AI optimization knowledge?

**