For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the complete content index is available at https://we0.ai/llms.txt, the full English article bundle is available at https://we0.ai/llms-full.txt, and this page is available as Markdown at https://we0.ai/articles/nvidia-nemotronlabs-voicechat-11b-open-full.md.
NVIDIA has released NemotronLabs VoiceChat 11B , an end-to-end real-time speech-to-speech model designed for natural full-duplex voice conve...
NVIDIA has released NemotronLabs VoiceChat 11B, an end-to-end real-time speech-to-speech model designed for natural full-duplex voice conversations.
Unlike the familiar pipeline that chains automatic speech recognition, a large language model, and text-to-speech together, VoiceChat handles streaming speech understanding and speech generation inside one unified architecture. The goal is to reduce the handoffs that normally add latency and make a voice assistant feel less natural.
The model can listen while a conversation is active, yield when the user interrupts, and resume naturally when the user finishes. NVIDIA reports a 448 ms smooth turn-taking latency on Full-Duplex-Bench 1.0 and about 480 ms interruption latency.
VoiceChat also introduces something especially relevant to production voice agents: live tool calling while the spoken conversation remains active. A separate output channel can emit tool-call instructions, while predefined acknowledgment messages help the agent avoid an awkward silent pause during an external API request.
The release is important, but it is still early. NVIDIA marks the model as ready for research purposes only, recommends substantial GPU memory for deployment, and documents several limitations around long conversations, noisy environments, reasoning, multi-tool use, and runaway speech.
A traditional real-time voice assistant often looks like this:
User speech
↓
ASR
↓
Text
↓
LLM
↓
Text response
↓
TTS
↓
Agent speech
VoiceChat moves these capabilities into a more tightly integrated model.
NVIDIA describes the 11B system as a hybrid Mamba/Transformer architecture composed of:
The user provides speech at 16 kHz. The output includes agent text, agent speech, and user transcription, with agent audio generated at 22.05 kHz.
A half-duplex assistant effectively treats conversation like a walkie-talkie: one side speaks, then the other side responds.
A full-duplex system can model both sides of the conversation continuously. That makes interruption handling, pauses, back-and-forth exchanges, and more natural turn taking possible.
NVIDIA’s published Full-Duplex-Bench 1.0 results include:
| Metric | VoiceChat 11B Result |
|---|---|
| Smooth turn-taking TOR | 0.82 |
| Smooth turn-taking latency | 448 ms |
| User interruption TOR | 1.0 |
| User interruption latency | 480 ms |
| User interruption GPT-4o score | 4.33 |
The 448 ms figure reported in the original news article comes from the smooth turn-taking benchmark. NVIDIA summarizes the model more broadly as delivering roughly 450 ms response latency.
Interruption handling is one of the biggest differences between a voice demo and a usable conversational agent.
VoiceChat is trained for conversational turn-taking directly. When a user starts talking in the middle of the model’s response, the system can stop its spoken turn and listen.
NVIDIA’s known-limitations documentation also says the behavior is not perfect in every situation. The model can still cut off a user on a mid-sentence pause, continue speaking after it should stop, start a new turn without new input, fall into a loop, or mishandle backchanneling.
VoiceChat 11B is NVIDIA’s first open full-duplex model with tool-calling support that is designed to maintain a natural spoken interaction while a tool is being used.
A voice agent frequently needs information that is not inside the model. For example:
What is the current status of my order?
The model may need to call an order-management API before it can answer.
VoiceChat supports acknowledgment messages for tools. A developer can define short phrases such as:
Sure, let me check that for you.
When the model decides to call the tool, the system can speak one of those phrases while the external request is handled.
A simplified flow looks like this:
User speaks
↓
VoiceChat responds
↓
Model determines a tool is needed
↓
Tool-call event sent to client
↓
Client executes the external function
↓
Tool result returned to VoiceChat
↓
VoiceChat resumes the spoken answer
NVIDIA recommends a maximum of about five tools per session because performance can decline when more tools are available.
The model also cannot yet reliably call several tools simultaneously.
Other limitations include incorrect tool selection, skipped tool calls, invented arguments, incorrectly spoken tool results, and answering from internal knowledge when it should have used a tool.
One especially important detail: although VoiceChat supports user interruption during normal conversation, the user cannot currently interrupt the agent during tool execution.
The reported AU Harness results are:
| Tool-Calling Category | Score |
|---|---|
| Simple | 58.5% |
| Multiple | 62.5% |
| Parallel | 42.5% |
| Parallel Multiple | 27.5% |
| Irrelevance | 89.6% |
| Average | 56.1% |
On Full-Duplex-Bench v3, NVIDIA reports:
| Metric | Score |
|---|---|
| Tool selection | 82.5% |
| Argument accuracy | 44.2% |
| Pass@1 | 33% |
These numbers show why production tool calls should still be guarded by application logic and argument validation.
NVIDIA reports that VoiceChat ranks #2 among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0.
The model is optimized for the trade-off between general intelligence and natural real-time conversation. NVIDIA explicitly warns that it may perform worse than the underlying Nemotron Nano v2 language model on knowledge, instruction following, safety, and reasoning tasks.
NVIDIA’s model card lists approximately 550,000 hours of audio training data.
The mixture includes both real and synthetic speech, alongside text data used for the language-model component. Named sources include Fisher, LibriVox, LibriTTS, HiFi-TTS, VCTK, PromptTTS, UltraChat, internal NVIDIA speech data, synthetic speech, Nemotron function-calling data, and PersonaPlex training data.
The current VoiceChat checkpoint does not support voice cloning.
NVIDIA’s repository says the released checkpoint uses one fixed voice. The purpose of this release is more focused: low-latency, full-duplex conversational behavior and tool-aware voice interaction.
NVIDIA’s current real-time deployment prerequisites specify:
NVIDIA GPU with at least 80 GB of VRAM
The documented container supports GPUs including NVIDIA A100, H100, RTX 6000 Pro, and B200. The broader model card also lists H200 and B100 compatibility.
For the optimized real-time container, NVIDIA currently requires x86_64, Linux, Docker, NVIDIA Container Toolkit, and compatible NVIDIA drivers.
The troubleshooting guide notes that the model itself uses approximately 66 GB of GPU memory.
As of August 11, the Hugging Face model page does not list an active inference provider for the checkpoint.
Instead, NVIDIA provides two main paths:
The real-time container packages CUDA, Triton, vLLM, and the complete VoiceChat inference stack and exposes a bidirectional WebSocket service.
Clone NVIDIA’s experimental VoiceChat branch:
git clone https://github.com/NVIDIA-NeMo/Speech.git
cd Speech
git switch nemotron-labs-voicechat
export NEMO_DIR="$(pwd)"
Describe your idea once, and We0 AI can generate a showcase site, pages, and CMS, then help you attract customers and traffic after launch.
One complete project generation for free registration
Best for trying one complete generation flow and seeing a first project draft quickly.
Create the environment:
conda create -y -n voicechat python=3.12
conda activate voicechat
pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0
pip install -e ".[all]"
pip uninstall -y nvidia-resiliency-ext
pip install transformers==4.56.0 tokenizers==0.22.0 lhotse==1.32.2 huggingface-hub==0.34.4 hf-xet==1.1.9 torchcodec==0.10.0 torch_audiomentations jinja2
pip install ninja packaging wheel einops
pip install --no-build-isolation --no-deps causal-conv1d==1.6.2.post1 mamba-ssm==2.3.2.post1
Download the checkpoint:
hf download nvidia/NVIDIA-NemotronLabs-VoiceChat-11B --local-dir /path/to/checkpoint
Run the sample:
conda activate voicechat
export NEMO_DIR=/path/to/Speech
python "$NEMO_DIR/examples/speechlm2/offline_voicechat_infer.py" --checkpoint /path/to/checkpoint --wav "$NEMO_DIR/examples/speechlm2/sample_audio/sample_general.wav" --output-dir /path/to/output
NVIDIA recommends adding enough trailing silence to custom input audio so the model has time to respond.
Download the model repository:
ngc registry model download-version nim/nvidia/nemotron-labs-voicechat:1.0.0
chmod -R 777 nemotron-labs-voicechat_v1.0.0
Launch the service:
docker run -it --rm --name=nemotron-labs-voicechat --runtime=nvidia --gpus '"device=0"' --shm-size=8GB -e NIM_HTTP_API_PORT=9000 -p 9000:9000 -v $(pwd)/nemotron-labs-voicechat_v1.0.0:/data/models --entrypoint /s2s/run_s2s_server.sh nvcr.io/nim/nvidia/nemotron-labs-voicechat:latest
Check readiness:
curl 'http://localhost:9000/v1/realtime/health'
The server then exposes a bidirectional WebSocket endpoint at:
ws://localhost:9000/v1/realtime
A simplified tool definition looks like this:
[
{
"name": "get_weather",
"description": "Get current weather for a city",
"ack_messages": [
"Sure, let me check that for you."
],
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string"
}
},
"required": ["city"]
}
}
]
The ack_messages field gives the system something natural to say while the tool is executing.
NVIDIA explicitly labels VoiceChat 11B as research purposes only.
The model was trained with audio context windows of no more than about two minutes, so longer conversational context may not be retained reliably.
Known issues include reasoning mistakes, hallucinations, degraded speech after several turns, repetition, garbled text, truncated speech, runaway continuation, self-talk, clarification loops, missing words in transcription, unreliable language switching, and poor behavior in noisy or highly reverberant environments.
Potential research and prototype scenarios include:
The agent can listen naturally, handle interruptions, call customer-service tools, and use acknowledgment messages while waiting for APIs.
A driver can interrupt an assistant without waiting for a full spoken response to finish. The current sensitivity to background noise means automotive deployment would require additional work.
A voice agent can gather an order, respond to corrections, and call product or inventory systems.
More natural turn-taking may help hands-free interfaces and voice-first applications, though accessibility deployments require careful user testing.
Organizations can experiment with internal tools and self-hosted speech interaction while keeping the model inside their own infrastructure.
The phrase “open source” deserves some precision.
The NeMo Speech code used by VoiceChat is licensed under Apache License 2.0.
The VoiceChat model materials use the OpenMDW 1.1 license.
For that reason, it is safer to describe VoiceChat 11B as an open model or open-weight model rather than assuming the model license is identical to the Apache-licensed software around it.
Teams planning redistribution or commercial use should review the OpenMDW license directly.
A useful evaluation plan should cover:
A voice agent that can call tools needs the same permission controls as any other autonomous agent.
NemotronLabs VoiceChat 11B is NVIDIA’s end-to-end real-time speech-to-speech model for full-duplex conversational AI. It combines streaming speech understanding, a Nemotron language-model backbone, speech generation, and a separate tool-calling channel.
NVIDIA reports 448 ms smooth turn-taking latency and about 480 ms interruption latency on Full-Duplex-Bench 1.0.
Yes, normal conversation supports barge-in and interruption handling. However, NVIDIA says users cannot currently interrupt the agent during tool execution.
Yes. The model can emit tool calls through a separate output channel, and developers can define acknowledgment messages that are spoken while the external tool is running.
NVIDIA’s real-time container requires a GPU with at least 80 GB of VRAM. The troubleshooting documentation says the loaded model uses roughly 66 GB.
NVIDIA currently documents self-hosted offline inference and an optimized real-time container. The Hugging Face model page does not currently list a hosted inference provider.
No. The released checkpoint uses a fixed voice and does not support voice cloning.
NVIDIA marks the model as research-only. Developers should evaluate its limitations around context length, reasoning, tool use, noisy audio, repetition, transcription, and runaway speech before production deployment.
NVIDIA NemotronLabs VoiceChat 11B combines speech understanding, language modeling, speech generation, interruption handling, and tool calling inside a unified full-duplex architecture. NVIDIA reports 448 ms smooth turn-taking latency and positions the model near the top of current open full-duplex voice benchmarks.
The most interesting engineering feature is tool use. A separate output channel can trigger external functions, while acknowledgment messages keep the conversation from falling silent during API calls.
The release is not yet a turnkey production service. Real-time deployment requires at least 80 GB of GPU memory, the current checkpoint has a fixed voice, hosted inference is not broadly available, and NVIDIA documents substantial limitations in longer sessions, reasoning, noisy audio, and tool execution.
VoiceChat 11B is best viewed as an open research platform for building and studying low-latency voice agents—not yet as a finished replacement for production contact-center or consumer voice APIs.
Start from one sentence and have a complete website in minutes.
