Realtime-Venus: Multimodal Conversational AI with Full-Duplex

September 19, 2026September 20, 2026

Sample outputs are available on the model card.

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Sample outputs are available on the model card.

Sample outputs are available on the model card.

Sample outputs are available on the model card.

Sample outputs are available on the model card.

Overview

inclusionAI has released “Realtime-Venus", a multimodal conversational system featuring asynchronous delegation and active full-duplex dialogue. It is a family of models capable of continuously perceiving input video, audio, and text while outputting text and audio waveforms over a shared causal timeline.

The repository contains two types of checkpoints: “Realtime-Venus-Omni", a 9B model that processes and integrates video and audio, and “Realtime-Venus-Audio", a 9B model specialized for voice conversation. Both support native full-duplex streaming dialogue, allowing the model to continue listening to external inputs even while speaking itself, discerning user backchannels, interruptions, and corrections to autonomously control response timing.

Specifications

Based on the published model card and technical documentation, the main specifications are as follows:

• Parameter count: 9B (for both Realtime-Venus-Omni and Realtime-Venus-Audio)
• Base architecture: MiniCPM-o 4.5 / Omni-Flow
• Language backbone: Qwen3-8B
• Visual encoder: SigLIP2 (used in Realtime-Venus-Omni; not used during inference in Realtime-Venus-Audio)
• Audio encoder: Whisper-Medium
• Audio generation decoder: Discrete S3 speech tokens and streaming Flow Matching decoder
• Input specifications:
• Realtime-Venus-Omni: Video, still images, audio, text – Realtime-Venus-Audio: Audio, text (input audio is 16kHz mono)
• Output specifications: Text and audio waveforms (generated audio during full-duplex streaming is output at 24kHz)
• Context length: Up to 40,960 tokens (input token upper limit is set to 32,768 tokens during offline chat)
• Weight precision: BF16
• Long video feature: Equipped with “Training-free long-video Memory" that retains and retrieves past visual moments without additional training (Realtime-Venus-Omni)
• License terms: Apache 2.0 (open-source license permitting commercial use and modification)

Performance and Quality

In the technical report published by the authors (arXiv:2609.13814), benchmark scores for video understanding, audio understanding/speech QA, and full-duplex dialogue are reported.

Realtime-Venus-Omni is reported to have achieved the highest scores in 6 out of 8 video benchmarks among models evaluated online. The specific scores are as follows:

• StreamingBench: 70.2%
• OVO-Bench: 64.7%
• Daily-Omni: 81.3%

As for Realtime-Venus-Audio, results leading comparison models are shown across 8 audio understanding and speech QA benchmarks:

• MMAU: 78.0%
• MMAU-Pro: 63.2%
• Llama Questions: 83.8%
• Speech CMMLU: 67.8%
• VoiceBench AlpacaEval: 4.81 (equivalent to the highest score among comparison targets)

Additionally, in “Full-Duplex-Bench v1.5", which evaluates full-duplex dialogue interaction performance, Realtime-Venus-Audio responded appropriately to user interruptions 75% of the time. Furthermore, the continuation rate of speech when users made simple backchannels reached 97%, the continuation rate during other-directed speech reached 88%, and the continuation rate under background speech such as environmental noise reached 86%. According to the authors, it is reported to have shown robustness exceeding the scores of Gemini 3.1 Live and GPT-4o in all three of these speech continuation metrics.

Strengths and Use Cases

Realtime-Venus features real-time processing of continuous audio and video streams and active response capabilities, making it powerful for multimodal conversations and automation tasks such as:

• Active Multimodal Monitoring and Response (Omni): Rather than simply waiting for prompts to be explicitly entered by the user, it can monitor camera feeds and microphone audio in real-time, allowing the model itself to autonomously initiate speech upon detecting specific events (e.g., boiling water, sports highlights, human movement, etc.).
• Advanced Full-Duplex Voice Conversation: Continuously receives microphone input even while speaking. It accurately distinguishes user backchannels, ambient noise, and other-directed speech to maintain conversation without interrupting speech. Meanwhile, it can instantly react to user interruptions, instruction changes, or question corrections to switch its speech output.
• Context Retention and Retrieval for Long Videos (Memory): Realtime-Venus-Omni is equipped with “Training-free long-video Memory" which requires no additional training. In long video streams of up to 40 minutes, it can archive and retrieve past important visual and audio moments, accurately reconstructing evidence to answer questions about past events.
• Asynchronous External Tool Execution (Delegation): Issues in-stream call requests in the middle of a conversation stream, allowing external tool execution and advanced tasks to be processed asynchronously in the background. Backend processing results can be seamlessly integrated into the conversation flow without blocking user interaction (uses “Realtime-Venus-Harness" for external task execution).
• Specialization in Voice Dialogue and Understanding (Audio): By using Realtime-Venus-Audio, you can execute direct full-duplex conversations, speech understanding, and speech QA from 16kHz mono audio without performing video processing. Output audio is generated at 24kHz.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch)

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

How to Get It

Model weights and custom code for Realtime-Venus can be downloaded in safetensors and ONNX formats from the Hugging Face repository (inclusionAI/Realtime-Venus). Gated access agreement prior to acquisition is not required.

Prerequisites for operation include Python 3.10, a CUDA environment, and FFmpeg for audio and video processing.

The acquisition and setup procedures are as follows:

1. Download the Repository:
   Use huggingface-cli or ModelScope to obtain the entire repository (including the subdirectories Realtime-Venus-Omni and Realtime-Venus-Audio).

huggingface-cli download inclusionAI/Realtime-Venus --local-dir.

2. Install Dependent Libraries:

python -m pip install -r Realtime-Venus-Omni/requirements.txt

3. Initialize the Model:
   Using Hugging Face Transformers’ AutoModel, load the model from the local subdirectory by specifying trust_remote_code=True.

• Example for loading Realtime-Venus-Omni:

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "./Realtime-Venus-Omni",
    trust_remote_code=True,
    local_files_only=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
).eval().cuda()

• Example for loading Realtime-Venus-Audio:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "./Realtime-Venus-Audio",
    trust_remote_code=True,
    local_files_only=True,
    fix_mistral_regex=True,
)
model = AutoModel.from_pretrained(
    "./Realtime-Venus-Audio",
    trust_remote_code=True,
    local_files_only=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    init_vision=False,
    init_audio=True,
    init_tts=True,
).eval().cuda()

When performing streaming full-duplex processing, call model.as_duplex() on the loaded model to switch to full-duplex mode. The runtime “Realtime-Venus-Harness" for asynchronous delegation and tool integration, as well as various sample codes, are provided in the official GitHub repository.

Sources

• https://huggingface.co/inclusionAI/Realtime-Venus
• https://arxiv.org/abs/2609.13814