LiveKit voice agents, self-hosted or on LiveKit Cloud
LiveKit carries live audio between a caller and an AI agent. We self-host it for our voice agent work, and this page covers the decisions that matter before an agent takes its first real call.
Reviewed by: Rahul Jaiswar, founder
What LiveKit is
LiveKit is an open-source server for real-time audio and video, with an Agents framework for building AI voice agents. You can run the server yourself or use LiveKit Cloud, the company’s hosted service.
How we use it
We run self-hosted LiveKit for real-time voice in our voice agents.
Self-hosted or LiveKit Cloud?
| Self-hosted | LiveKit Cloud | |
|---|---|---|
| Where audio runs | A server in the region you choose | LiveKit’s infrastructure |
| Who keeps it running | We deploy and host it in an account you own, and cover after launch follows the support level agreed in the scope of work | LiveKit |
| Features | The open-source server | Extra hosted features; some, such as its noise cancellation, may not run self-hosted |
| Bill | Server cost you control | Usage-based |
| Phone calls | Needs LiveKit’s SIP service running alongside the server | Telephony offered as part of the hosted service |
When it fits
- A voice agent in a browser or on phone calls: answering routine questions, taking details, or following up.
- Audio, recordings and transcripts that must stay on servers you pick. Self-hosting LiveKit achieves that only if speech-to-text, text-to-speech and the model also run on your servers.
- Calls that may pass from the agent to a person partway through.
When it doesn’t
- A fixed menu (“press 1 for dispatch”) needs a phone system, not an AI voice stack.
- Recording calls, and nothing more. Telephony providers can usually do that on their own. For phone lines, see Telnyx.
Which language the agent speaks is a project decision. We test a model in the language your callers use before committing to it, so an unusual request is never ruled out by a list.
Five questions for your build team
- Self-hosted or Cloud, and why for our case?
- Where will the server sit relative to our callers?
- Where are recordings and transcripts stored, and who can open them?
- What happens to a caller if the agent process crashes mid-call?
- How does a call pass to a person, and what does that person see?
Before a real call
- Transcripts can quietly stop saving. Some conversation items are not speech, and code that treats them as speech can fail with no visible error. Filter conversation items to messages before saving the transcript, and read the transcript of a real test call, not only the logs.
- Idle memory. Every warm agent process loads speech models, so set how many stay warm.
- Microphones need a secure page. Browsers grant the microphone only over HTTPS, and that page then needs a secure WebSocket back to the server.
- Media ports. Audio uses its own ports, which have to be opened and tested from outside the server.
- Strict firewalls. Some office and mobile networks block those ports. Configure TURN over TLS on port 443 as a fallback so those callers still connect.
Planning the handover to a person
Decide before building what the agent passes on when a caller asks for a human: a live transfer, a callback, or a written summary for your team. Then decide what that person sees on screen when the call reaches them, so the caller doesn’t have to start again. Test the handover on a real call as carefully as the agent’s answers.
Related
Planning a voice agent on LiveKit?
Start with the calls it should take, and whether you already have a LiveKit account.