Orion-26B-A4B-v1.1 GGUF Quants Released by bartowski

September 18, 2026

Orion-26B-A4B-v1.1 GGUF Quants Released by bartowski

At a Glance

Item Value
Repository bartowski/TheDrummer_Orion-26B-A4B-v1.1-GGUF
Published 2026-09-14
Formats GGUF
Source type Primary source (the publisher itself)

Values determined by this site’s code at collection time. Dates are JST.

Overview

It is reported that bartowski has released GGUF quantization files for the multimodal model Orion-26B-A4B-v1.1. As some information included could not be officially verified at the time of writing, caution is advised when deploying or testing the model in practice. It supports text, image, and audio inputs, and a variety of quantization files are provided for local execution.

Specifications

  • Parameters: 26B
  • Input Support: Text, image, audio (requires mmproj file)
  • Speculative Decoding: None
  • imatrix: Supported
  • Architecture: Gemma4ForConditionalGeneration

Performance

Regarding the performance of the base model, according to the model card, a low temperature setting (such as 0.9) is optimal, and Gemma’s thinking capabilities are reported to be very sharp in roleplay (RP). Below is a comparison table of per-tensor layouts for the published quantized versions.

→ Scroll horizontally to see all columns

Quant Size Body bits/weight File bits/weight Body kept at base type
Q6_K_L 24.52GB 7.38 7.60 51 %
Q6_K 23.79GB 7.03 7.37 71 %
Q6_K_S 23.14GB 6.71 7.17 90 %
Q5_K_M 20.01GB 5.92 6.20 71 %
Q5_K_S 18.40GB 5.70 5.70 90 %
Q4_K_L 18.21GB 5.19 5.64 50 %
Q4_K_M 17.97GB 5.07 5.57 70 %
Q4_K_S 16.05GB 4.70 4.98 90 %
IQ4_NL 15.63GB 5.07 4.85 70 %
Q3_K_L 14.80GB 4.39 4.59 50 %
IQ3_M 14.34GB 4.17 4.45 50 %
IQ4_XS 14.28GB 4.42 4.43 90 %
Q3_K_M 13.79GB 4.01 4.27 70 %
IQ3_XS 12.84GB 3.57 3.98 90 %
IQ3_XXS 12.61GB 3.46 3.91 70 %
Q3_K_S 12.37GB 3.56 3.84 90 %
Q2_K 11.61GB 3.19 3.60 70 %
IQ2_M 10.90GB 2.89 3.38 70 %
IQ2_S 10.31GB 2.61 3.20 70 %
IQ2_XS 9.90GB 2.41 3.07 90 %

(Table truncated. Subsequent rows omitted)

Furthermore, the comparative verification results between the computed layout and the standard layout are shown below.

→ Scroll horizontally to see all columns

Quant Computed layout KLD Standard layout KLD Ratio at equal size Size vs standard file
Q4_K_M 0.8086 ± 0.0113 1.1462 ± 0.0132 0.88× +5.5 %
Q3_K_M 1.2642 ± 0.0149 2.9859 ± 0.0241 0.51× +5.9 %
IQ2_XS 2.6730 ± 0.0206 2.9355 ± 0.0210 0.91× −2.3 %

From these tables, it is confirmed that quantization files adopting the computed layout, such as Q4_K_M and Q3_K_M, keep the Kullback-Leibler divergence (KLD) lower compared to the standard layout, resulting in smaller differences from the unquantized model and maintained quality. Q4_K_M in particular shows a favorable balance.

Strengths and Use Cases

This model supports image and audio input in addition to text, and is intended for conversational use and roleplay (RP) applications. Inheriting the characteristics of the base model, it is well-suited for interactive tasks leveraging multimodal inputs.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 25.8B parameters (taken from the base model TheDrummer/Orion-26B-A4B-v1.1)

Your VRAM Quantization File size Est. memory needed
8GB (RTX 4060 / 3060 Ti, etc.) F16 1.1GB 1.3GB
12GB (RTX 4070 / 3060 12GB, etc.) IQ2_S 9.6GB 11.5GB
16GB (RTX 5060 Ti 16GB / 4060 Ti 16GB, etc.) IQ3_M 13.4GB 16.0GB
24GB (RTX 4090 / 3090, etc.) Q5_K_M 18.6GB 22.4GB
32GB (RTX 5090, etc.) Q8_0 25.0GB 30.0GB
80GB class (A100 / H100) BF16 48.1GB 57.8GB

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

How to Get It

  • Distribution Format: GGUF
  • Supported Engines: llama.cpp, LM Studio, koboldcpp, ramalama, Jan AI, Text Generation Web UI, LoLLMs, Atomic Chat
  • Example Download Command:
hf download bartowski/TheDrummer_Orion-26B-A4B-v1.1-GGUF --include "TheDrummer_Orion-26B-A4B-v1.1-Q4_K_M.gguf" --local-dir./

When utilizing multimodal inputs, you will also need to prepare the corresponding multimodal projector file (mmproj-TheDrummer_Orion-26B-A4B-v1.1-f16.gguf or mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf).

Related Articles

Sources

New Modelsbartowski,GGUF,llama.cpp,Orion-26B-A4B-v1.1,マルチモーダル,確認済み