NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF Multimodal Model: 4GB+ VRAM

NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF Multimodal Model: 4GB+ VRAM

At a Glance

Item Value
Repository bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF
Published 2026-10-25
Formats GGUF
Source type Unverified (not confirmed by a primary source)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

It is reported that bartowski has released a GGUF quantized repository, bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF, on Hugging Face for what is said to be an NVIDIA model named “NVIDIA-Nemotron-3.5-Bolt-E4B". However, note that as of this writing, no official announcement has been confirmed for this model or its purported base model, nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, making this unverified information.

According to the model card description, the base model is an 8B parameter scale model capable of multimodal processing supporting text, image, and audio inputs. This repository was quantized using llama.cpp release b11845 leveraging imatrix (importance matrix), and is described as being configured to use multimodal capabilities when combined with a multimodal projector file.

Specifications

  • Parameters: 8B (source checkpoint)

Performance

While the model card does not list measured scores for general task benchmarks (such as MMLU or HumanEval), measurement results comparing the Kullback-Leibler Divergence (KLD) against the unquantized model are published as local verification data from the publisher.

The comparison results between the computed tensor-wise layout (Computed layout) and the standard llama.cpp layout (Standard layout) listed in the model card are as follows:

→ Scroll horizontally to see all columns

Quant Computed layout KLD Standard layout KLD Ratio at equal size Size vs standard file
Q4_K_M 0.0911 ± 0.0014 0.1243 ± 0.0019 0.28× −4.5 %
Q3_K_M 0.3504 ± 0.0041 0.9233 ± 0.0085 0.31× −17.2 %
IQ2_M 0.9538 ± 0.0086 1.1377 ± 0.0099 0.70× −20.8 %

From these measurement results, the following tendencies can be observed.

According to the publisher’s explanation, this verification adopts a method (Computed layout) that optimizes precision allocation per tensor according to model sensitivity. Looking at the ratio at equal size for Kullback-Leibler Divergence, which is an indicator of information loss relative to the unquantized model, it is 0.28× for Q4_K_M, 0.31× for Q3_K_M, and 0.70× for IQ2_M, reporting that KLD is kept significantly lower compared to the standard layout across all quantization types.

In particular, the degree of deviation from the standard layout is greatly improved in Q4_K_M and Q3_K_M, suggesting that degradation is mitigated by intensively allocating precision to high-sensitivity weights even at equivalent file sizes.

Note that these are indicators showing relative degrees of degradation due to quantization methods, and numerical values for the model’s original inference capability or actual performance on language and multimodal tasks are not stated in the documentation.

Strengths and Use Cases

According to the model card description and configuration tags, this model is described as excelling in multimodal processing (any-to-any) supporting image and audio inputs in addition to text. When combined with the accompanying multimodal projector file (mmproj), it is configured to run dialogue processing involving image and audio recognition in a local environment.

Additionally, the prompt format specifications explicitly detail structures handling tool definitions and invocations (Tool-calling / Function-calling). The calibration corpus for creating the imatrix (importance matrix) also used dialogue data containing tool calls and logical reasoning in addition to general prose, and the model card explains that calibration was performed using the exact same chat template and special tokens as during inference.

From this information, it is thought that this model is anticipated for use in applications such as:

  • Local multimodal chats and QA incorporating images and audio
  • Agentic tasks calling external APIs or scripts based on tool definitions
  • Text generation tasks accompanied by daily conversation and logical reasoning

Note that since this model is considered an unverified model whose official announcement could not be confirmed as of this writing, future verification regarding actual dialogue quality and tool utilization stability is awaited.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch)

Your VRAM Quantization File size Est. memory needed
4GB (laptop iGPU / phone class) IQ2_M 2.9GB 3.5GB
8GB (RTX 4060 / 3060 Ti, etc.) Q6_K_L 6.2GB 7.4GB
12GB (RTX 4070 / 3060 12GB, etc.) Q8_0 7.5GB 9.0GB
24GB (RTX 4090 / 3090, etc.) BF16 14.0GB 16.8GB

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

Runs in Ollama, LM Studio and llama.cpp as-is.

It is distributed in GGUF, so no conversion is needed.

Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists.

Our Own Measurements

Measurements We Did Not Take

This model does not state a license, so we cannot confirm permission to use it. Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation).

How to Get It

Prolific in GGUF format, this model is available from bartowski’s repository on Hugging Face. The repository is published as non-gated (gated: false), so prior access approval applications are not required for use.

Supported Engines

It is noted that llama.cpp release b11845 was used for the quantization work, and the model is guided to be compatible with various local inference tools and frontends including llama.cpp and the following from that release onward:

  • llama.cpp
  • LM Studio
  • koboldcpp
  • ramalama
  • Jan AI
  • Text Generation Web UI
  • LoLLMs
  • Atomic Chat

Download Steps

To download individual GGUF files using the Hugging Face CLI, run the following commands.

First, install the CLI tool.

pip install -U "huggingface_hub[cli]"

Next, download by specifying Q4_K_M or similar as guided in the recommended settings.

hf download bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF --include "NVIDIA-Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf" --local-dir./

Execution Method and Using the Multimodal Projector

When running directly via the official llama.cpp launcher, the following one-liner and commands are posted on the model card:

curl -LsSf https://llama.app/install.sh | sh
llama-server -hf bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF:Q4_K_M

After launching llama-server, you can access the built-in chat Web UI by navigating to http://localhost:8080 by default.

When utilizing multimodal features for inputting images and audio, using the multimodal projector files bundled in this repository (mmproj-NVIDIA-Nemotron-3.5-Bolt-E4B-bf16.gguf or mmproj-NVIDIA-Nemotron-3.5-Bolt-E4B-f16.gguf) is required. When loading the model using the -hf option, the projector file is automatically fetched, but when loading files manually, you must specify --mmproj in the startup arguments to pass the projector file.

Quantization Type Selection Guidelines

A wide variety of quantization files are available within the repository. The publisher’s model card suggests the following selection criteria:

  • Standard Selection: Q4_K_M, which offers an excellent balance, is recommended as the general default.
  • Quality-First: For those seeking higher precision, options like Q5_K_M, Q6_K_L, and highest-precision Q8_0 are listed.
  • Low-Bit Quantization (I-quant / K-quant): It is explained that when running sub-Q4 low-bit quantizations in GPU environments (NVIDIA cuBLAS or AMD rocBLAS), size-efficient IQ quantizations (IQ3_M, IQ2_M, etc.) are suitable. While these can also run on CPU, there is a trade-off where inference speeds are lower compared to equivalent K-quants.
  • Notes on CPU Inference: It is explained that recent versions of llama.cpp feature runtime online repacking for ARM and AVX, enabling fast inference without requiring special CPU-only files for Q4_0, IQ4_NL, and most K-quants.

Related Articles

What to Read Next

Sources

This article contains unverified information. We will append an update note once it is confirmed by a primary source.