{"id":12732,"date":"2026-11-17T11:01:34","date_gmt":"2026-11-17T02:01:34","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=12732"},"modified":"2026-11-17T11:10:49","modified_gmt":"2026-11-17T02:10:49","slug":"nvidia-cosmos3-h-dreams-surgical-world-model","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-cosmos3-h-dreams-surgical-world-model\/","title":{"rendered":"Cosmos3-H-Dreams Video Generation Model: 48GB+ VRAM, File List"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Dreams\">nvidia\/Cosmos3-H-Dreams<\/a><\/td>\n<\/tr>\n<tr>\n<td>Publisher guide<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-nvidia-en\/\">NVIDIA: models and licenses<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-10-19<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>nvidia-open-model-license<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/format-safetensors-en\/\">safetensors<\/a><\/td>\n<\/tr>\n<tr>\n<td>Paper<\/td>\n<td><a href=\"https:\/\/arxiv.org\/abs\/2608.24199\">arXiv:2608.24199<\/a>, <a href=\"https:\/\/arxiv.org\/abs\/2506.08009\">arXiv:2506.08009<\/a>, <a href=\"https:\/\/arxiv.org\/abs\/2405.14867\">arXiv:2405.14867<\/a><\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>On October 20, 2026, NVIDIA released &#8220;Cosmos3-H-Dreams&#8221;, a real-time action-conditioned surgical world model designed for the &#8220;CMR Surgical Versius&#8221; robotic surgery system. This model was developed by fine-tuning and distilling the foundational model for embodied AI, &#8220;Cosmos3-H-Embodied-Nano&#8221; (with Cosmos3-Nano as its base model), using clinical cholecystectomy data.<\/p>\n<p>This model accepts a single endoscopic camera frame, recent robotic operation history, and future kinematic and control command sequences for both left and right arms as inputs. Based on these inputs, it autoregressively and in real-time streams generated endoscopic videos of the surgical field corresponding to the operations. Its primary roles include evaluating robot policies in a simulation environment before transitioning to physical systems, interactive surgical field exploration, and the synthesis of surgical data.<\/p>\n<h2>Specifications<\/h2>\n<p>The specifications of Cosmos3-H-Dreams as listed in the model card are as follows:<\/p>\n<ul>\n<li><strong>Parameter Count:<\/strong> Approximately 1.5 \u00d7 10^10 (15.17B)<\/li>\n<li><strong>Architecture:<\/strong> Mixture-of-Transformers (MoT). It consists of a Qwen3-VL-8B-Instruct inference tower and a latent diffusion generation expert equipped with shared multi-modal attention. It operates on the Wan2.2 VAE latent space (compressed 4x temporally and 16&#215;16 spatially) and features action embedding and projection layers for kinematic conditioning. It is distilled from a bidirectional teacher model into a block-causal 4-step autoregressive student model equipped with a KV cache.<\/li>\n<li><strong>Input Specifications:<\/strong><\/li>\n<li>Image: A single 2D endoscopic RGB frame (recommended resolution 832\u00d7480, 16:9 aspect ratio recommended) &#8211; Action Vector: 28-dimensional (1D sequence). 14 dimensions are assigned to each of the left and right arms (relative translation 3, relative 6D rotation 6, jaw open\/close 1, electrosurgical energy activation 1, thumbstick button 1, thumbstick axes 2). Past action history accepts up to 16 steps (1.6 seconds, at 10 Hz), and future control commands accept up to 72 steps.<\/li>\n<li><strong>Output Specifications:<\/strong><\/li>\n<li>Format: MP4 video (3D: height, width, time) &#8211; Resolution \/ Framerate: 832\u00d7480, 10 Hz &#8211; Generation Unit: Generates 72 frames (7.2 seconds) by default and streams output in units of 4 frames (0.4 seconds). It also supports longer trajectory generation by continuing or re-anchoring the autoregressive loop.<\/li>\n<li><strong>Recommended Inference Steps:<\/strong> 4 steps per latent frame (distilled student model)<\/li>\n<li><strong>Licensing Terms:<\/strong> Supports both commercial and non-commercial use (Ready for commercial or non-commercial use). The nvidia-open-model-license applies.<\/li>\n<\/ul>\n<h2>Performance and Quality<\/h2>\n<p>In the Cosmos3-H-Dreams model card, inference speed benchmark results on NVIDIA hardware environments are published. The table of published measured values is as follows:<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>GPU<\/th>\n<th>Resolution (W x H)<\/th>\n<th style=\"text-align: right;\">Diffusion steps<\/th>\n<th style=\"text-align: right;\">Inference FPS<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RTX PRO 6000 Blackwell Workstation Edition<\/td>\n<td>832 x 480<\/td>\n<td style=\"text-align: right;\">4<\/td>\n<td style=\"text-align: right;\">30.0<\/td>\n<\/tr>\n<tr>\n<td>RTX PRO 6000 Blackwell Workstation Edition<\/td>\n<td>832 x 480<\/td>\n<td style=\"text-align: right;\">2<\/td>\n<td style=\"text-align: right;\">43.2<\/td>\n<\/tr>\n<tr>\n<td>RTX PRO 6000 Blackwell Max-Q Workstation Edition<\/td>\n<td>832 x 480<\/td>\n<td style=\"text-align: right;\">4<\/td>\n<td style=\"text-align: right;\">20.2<\/td>\n<\/tr>\n<tr>\n<td>RTX PRO 6000 Blackwell Max-Q Workstation Edition<\/td>\n<td>832 x 480<\/td>\n<td style=\"text-align: right;\">2<\/td>\n<td style=\"text-align: right;\">31.2<\/td>\n<\/tr>\n<tr>\n<td>DGX Spark<\/td>\n<td>512 x 288<\/td>\n<td style=\"text-align: right;\">2<\/td>\n<td style=\"text-align: right;\">16.9<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>According to the measurements published by the release origin, on the RTX PRO 6000 Blackwell Workstation Edition, it achieves 30.0 FPS at the standard settings of 832\u00d7480 resolution and 4 diffusion steps, which improves up to 43.2 FPS when the steps are reduced to 2. On the power-saving RTX PRO 6000 Blackwell Max-Q Workstation Edition, it records 20.2 FPS with 4-step execution and 31.2 FPS with 2-step execution at the same resolution. On the other hand, in the DGX Spark environment, even when running with the resolution reduced to 512\u00d7288 and 2 steps, it remains at 16.9 FPS, demonstrating that significant differences arise in the obtained inference frame rate depending on the execution hardware configuration and input resolution.<\/p>\n<p>Regarding quality, validation was conducted using CMR Surgical Versius cholecystectomy data recorded in clinical settings (a dataset of 4,792 episodes, downsampled from approximately 17 million frames of 1080p and 60 Hz to 10 Hz). It has been confirmed that endoscopic videos of the surgical field change in accordance with the kinematics and control commands of the operator&#8217;s two arms, and it is described as having the quality to respond interactively to operation trajectories output by human operators or trained policy models.<\/p>\n<h2>Strengths and Use Cases<\/h2>\n<p>Cosmos3-H-Dreams excels at action-conditioned real-time surgical simulation specialized for laparoscopic cholecystectomy using the surgical support robot &#8220;CMR Surgical Versius&#8221;. A major feature of this model is that it goes beyond passive video generation, dynamically generating endoscopic images in response to continuous control inputs provided by the operator.<\/p>\n<p>The model card lists the following primary intended use cases:<\/p>\n<ul>\n<li><strong>Policy Evaluation Prior to Physical Robot Migration:<\/strong> Animal experiments and cadaver-based validation using real hardware impose heavy burdens in terms of cost and reproducibility. By using this model, performance evaluation of reinforcement learning policies or VLA (Vision-Language-Action) policies for surgical robots can be performed at low cost in a simulation environment.<\/li>\n<li><strong>Real-time and Interactive Surgical Field Exploration:<\/strong> Instantly responds to inputs using an operator&#8217;s teleoperation device, keyboard, or headsets such as Meta Quest, allowing interactive observation and verification of environmental changes in the surgical field.<\/li>\n<li><strong>Generation of High-Quality Synthetic Surgical Data:<\/strong> Enables large-scale creation of synthetic datasets pairing consistent endoscopic video with control commands for specific operational trajectories and approaches that are difficult to collect solely from clinical data.<\/li>\n<\/ul>\n<p>While the base model Cosmos3-H-Embodied-Nano was a basic action-conditioned model targeting multiple types of surgical robots, Cosmos3-H-Dreams undergoes additional training specialized for actual cholecystectomy data by the CMR Versius. By limiting the surgical field to views showing grasping forceps and an electrosurgical hook on the left and right arms, and precisely reflecting 28-dimensional operation inputs, it aims to provide a high-fidelity simulation environment adapted to actual clinical procedures.<\/p>\n<p><!-- lmw:hardware --><\/p>\n<h2>Hardware Requirements<\/h2>\n<p><strong>Estimated requirements (calculated by Local Model Watch)<\/strong> \u2014 15.8B parameters (taken from the base model nvidia\/Cosmos3-Nano)<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Your VRAM<\/th>\n<th>Quantization<\/th>\n<th>File size<\/th>\n<th>Est. memory needed<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>48GB (RTX 6000 Ada \/ A6000, etc.)<\/td>\n<td>BF16<\/td>\n<td>32.5GB<\/td>\n<td>39.0GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><strong>Inference engine support<\/strong> (architecture name matched against each project&#8217;s own model registry in its source code, checked 2026-11-17): llama.cpp: not registered, vLLM: not registered, MLX (mlx-lm): not registered. &#8220;Not registered&#8221; means the name is absent from that registry today, not that the model cannot run.<\/p>\n<p><em>Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model&#8217;s authors. Compare with other models in our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a>. What the quantization names mean: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">glossary<\/a>.<\/em><\/p>\n<p><!-- \/lmw:hardware --><\/p>\n<p><!-- lmw:runnability --><\/p>\n<h2>Can You Run It Locally?<\/h2>\n<p>The publisher distributes this model as safetensors.<\/p>\n<p><strong>License \u2014 <code>nvidia-open-model-license<\/code>:<\/strong> A custom license from the publisher. Check the original terms directly, including whether commercial use is permitted.<\/p>\n<p><em>Compiled by this site&#8217;s code from the published formats and the license field. License summaries are not legal advice \u2014 check the publisher&#8217;s original terms before relying on them.<\/em><\/p>\n<p><!-- \/lmw:runnability --><\/p>\n<p><!-- lmw:lab --><\/p>\n<h2>Our Own Measurements<\/h2>\n<h3>Measurements We Did Not Take<\/h3>\n<p>We have not confirmed that this site meets the commercial-use terms of this model&#8217;s license (<code>nvidia-open-model-license<\/code>). Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation).<\/p>\n<p><!-- \/lmw:lab --><\/p>\n<p><!-- lmw:files --><\/p>\n<h2>Distributed Files<\/h2>\n<p><em>Weight files published in <a href=\"https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Dreams\/tree\/main\">nvidia\/Cosmos3-H-Dreams<\/a>, listed by this site from the Hugging Face API. Sizes are the actual file sizes.<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>File<\/th>\n<th>Size<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>checkpoints\/exp15\/c3-h-d_2-step_model_bf16.safetensors<\/code><\/td>\n<td>30.35GB<\/td>\n<\/tr>\n<tr>\n<td><code>checkpoints\/exp15\/c3-h-d_4-step_model_bf16.safetensors<\/code><\/td>\n<td>30.35GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:files --><\/p>\n<h2>How to Get It<\/h2>\n<p>Model weights for Cosmos3-H-Dreams are published on Hugging Face in safetensors format. There are no access restrictions (Gated), and developers who agree to the terms of use can download and use them.<\/p>\n<p>Specifications regarding the model execution environment and available tools are as follows:<\/p>\n<ul>\n<li><strong>Supported Repositories:<\/strong> Weight files are provided at the Hugging Face repository <code>nvidia\/Cosmos3-H-Dreams<\/code>. The inference codebase is published on GitHub at <a href=\"https:\/\/github.com\/isaac-for-healthcare\/Cosmos3-H-Dreams\">Cosmos3-H-Dreams<\/a>.<\/li>\n<li><strong>Streaming Inference Runtime:<\/strong> <code>Cosmos3-H-Dreams<\/code> and <a href=\"https:\/\/github.com\/NVIDIA\/flashdreams\">FlashDreams<\/a> are provided as inference libraries to realize real-time delivery. Static-shape KV cache, <code>torch.compile<\/code>, CUDA Graphs, the LightTAE lightweight decoder, and operation delivery stacks via keyboards and WebXR (Meta Quest) are integrated.<\/li>\n<li><strong>Supported Hardware and OS:<\/strong> GPUs with NVIDIA Blackwell and NVIDIA Hopper architectures are supported. The only verified operating system is Linux.<\/li>\n<li><strong>Operation Precision:<\/strong> Training and native execution are performed in BF16 (Brain Floating Point 16) mixed precision, and weights are exported in FP32 format. No official operational verification has been conducted for other quantization formats or low-precision settings at this time.<\/li>\n<\/ul>\n<p>When running inference, there is no need to separately prepare the complete Cosmos 3 training framework; it can be operated using only the lightweight streaming inference stack.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-qwen-image-nvfp4-released\/\">Qwen-Image-NVFP4 4-Bit Quantized Image Generation Model: 48GB+ VRAM<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-metrictrail-release\/\">MetricTrail: File List<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-releases-kimodo-soma-rp-v1-2\/\">NVIDIA Releases Kimodo-SOMA-RP-v1.2 for 3D Motion Generation<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-releases-ardy-soma-rp-30fps-kimodorep-horizon60\/\">ARDY-SOMA-RP-30FPS-KimodoRep-Horizon60: File List<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Find models by VRAM<\/strong> (This model needs at least 48GB) \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a><\/li>\n<li><strong>Formats this model is available in<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/format-safetensors-en\/\">Safetensors format guide and models<\/a><\/li>\n<li><strong>Learn about the publisher<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-nvidia-en\/\">NVIDIA: models, licenses and articles<\/a><\/li>\n<li><strong>Other models for the same task<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/models-by-task-en\/#task-video\">Other video generation models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Dreams\">https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Dreams<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/isaac-for-healthcare\/Cosmos3-H-Dreams\">https:\/\/github.com\/isaac-for-healthcare\/Cosmos3-H-Dreams<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2608.24199\">https:\/\/arxiv.org\/abs\/2608.24199<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Embodied-Nano\">https:\/\/huggingface.co\/nvidia\/Cosmos3-H-Embodied-Nano<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/nvidia\/Cosmos3-Nano\">https:\/\/huggingface.co\/nvidia\/Cosmos3-Nano<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>NVIDIA has released Cosmos3-H-Dreams, a real-time action-conditioned surgical world model for the CMR Surgical Versius robot system.<\/p>\n","protected":false},"author":1,"featured_media":12731,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[203],"tags":[3848,3850,3852,695,1547,959,2663,3854],"class_list":["post-12732","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-image-video-and-audio","tag-cosmos3-h-dreams-en","tag-cosmos3-nano-en","tag-mot-en","tag-nvidia-en","tag-verified","tag--en","tag-ai-en"],"lang":"en","translations":{"en":12732,"ja":12730},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12732","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=12732"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12732\/revisions"}],"predecessor-version":[{"id":17885,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12732\/revisions\/17885"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/12731"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=12732"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=12732"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=12732"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}