Aion-1.0-Instruct-2.3B-GGUF 2.3B Text Generation Model: 4GB+ VRAM

Aion-1.0-Instruct-2.3B-GGUF 2.3B Text Generation Model: 4GB+ VRAM

At a Glance

Item Value
Repository AtomicChat/Aion-1.0-Instruct-2.3B-GGUF
Publisher guide Microsoft: models and licenses
Published 2026-11-19
License mit
Formats GGUF
Source type Unverified (not confirmed by a primary source)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

It is reported that the GGUF version of “Aion 1.0 Instruct 2.3B," which is said to be Microsoft’s next-generation Windows on-device SLM (Small Language Model), has been released by AtomicChat. Note that this information has not been officially confirmed. This model is stated to be a public release aimed at introduction and ensuring transparency, ahead of the scheduled rollout of an optimized version as a Windows OS SLM by the end of January 2027.

Specifications

  • Parameters: 2.29B (including 0.41B for tied embedding / head)
  • Layers: 28 (21 sliding-window layers (2048) + 7 full attention layers)
  • Attention: GQA, 24 query heads, 8 KV heads, head dim 128, QK RMSNorm
  • Hidden / FFN: 2048 / 8192 (SwiGLU)
  • RoPE: theta 32000; YaRN x16 from 2048 on the full-attention layers
  • Context length: 32,768 tokens
  • Vocabulary: 200,029 (o200k)

Performance

According to the publisher’s measurements, the performance of the original model microsoft/Aion-1.0-Instruct-2.3B is as follows:

Category Benchmark Score
Knowledge MMLU 55.15
Knowledge MMLU-Pro 33.35
Knowledge MMMLU 45.51
Knowledge GPQA 28.18
Math GSM8K 74.91
Math GSMPlus 49.56
Math MATH-500 48.00
Instruction following IFEval 83.90
Instruction following IFBench 32.00
Instruction following Multi-IF 64.70
Translation chrF++ 43.18

From this table, it can be seen that the original model shows high scores, such as 74.91 on GSM8K, which measures the ability to solve math word problems, and 83.90 on IFEval, which measures the ability to follow format instructions. On the other hand, it scores 28.18 on GPQA, which handles difficult scientific questions, a relatively low figure compared to other metrics. Overall, this suggests that the model is designed to deliver high performance in specific tasks despite being lightweight.

Additionally, regarding the quantization precision of the released GGUF version, the following comparison table is provided:

file mean KLD same top token
BF16 0 100%
Q8_0 0.0029 96.6%
AD-Q5_K_M 0.0062 95.1%
AD-Q4_K_M 0.0131 93.0%
AD-IQ3_S 0.0705 84.1%

According to this table, AD-Q4_K_M, recommended by AtomicChat, maintains a “same top token" ratio (the proportion of selecting the same next token as the original model) of 93.0% compared to the original BF16, achieving weight reduction while suppressing degradation due to quantization. KLD (Kullback-Leibler divergence), an indicator of how far the quantized distribution deviates from the original distribution, is kept relatively low at 0.0131 for AD-Q4_K_M. For the most lightweight AD-IQ3_S, the probability of selecting the same token as the original model drops to 84.1%.

Strengths and Use Cases

The original model microsoft/Aion-1.0-Instruct-2.3B is primarily trained for general-purpose text scenarios such as summarization, rewriting, document question answering, and chatting. It supports 8 languages: English, German, Spanish, Portuguese, Italian, French, Chinese, and Japanese. Furthermore, equipped with a 32K context length, it is capable of processing multi-page documents or long email threads locally. However, the original model’s model card points out that it is not specifically trained for world knowledge and has limitations in advanced multi-step reasoning.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 2.3B parameters (taken from the base model microsoft/Aion-1.0-Instruct-2.3B)

Your VRAM Quantization File size Est. memory needed
4GB (laptop iGPU / phone class) Q8_0 2.3GB 2.7GB
8GB (RTX 4060 / 3060 Ti, etc.) BF16 4.3GB 5.1GB

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

Runs in Ollama, LM Studio and llama.cpp as-is.

It is distributed in GGUF, so no conversion is needed.

License — mit (Commercial use allowed): Permits commercial use, modification and redistribution, provided the copyright notice and license text are retained.

Compression: the Q8_0 build measures 8.63 bits per weight — about 54% the size of the original 16-bit weights, calculated by this site from the actual file sizes.

Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists. License summaries are not legal advice — check the publisher’s original terms before relying on them.

Recent Models in the Same Size Class

Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.

→ Scroll horizontally to see all columns

Model Parameters Smallest VRAM tier License Our article
ggml-org/LFM3-1.7B-GGUF 1.7B 4GB lfm1.0 LFM3-1.7B-GGUF Text Generation Model: Our Test Answers, 4GB+ VRAM (2026-11-18)
openbmb/MiniCPM-V-4.7-35B-A3B-DSpark-gguf 389M 4GB apache-2.0 MiniCPM-V-4.7-35B-A3B-DSpark-gguf Draft Model for Speedup: 4GB+ VRAM (2026-11-17)
openbmb/MiniCPM-V-4.7-1B-DSpark-gguf 214M 4GB apache-2.0 MiniCPM-V-4.7-1B-DSpark-gguf Text Generation Model: 4GB+ VRAM (2026-11-17)
IFM/Jais-3-0.9B-GGUF 1.1B 4GB apache-2.0 Jais-3-0.9B-GGUF Bilingual Dense Model: Our Test Answers, 4GB+ VRAM (2026-11-17)
nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 4.0B 4GB openmdw-1.1 NVIDIA Releases Dory1.0-4B: A Recurrent 4B LLM with Dynamic Loops (2026-11-17)

How to Get It

The GGUF version files released by AtomicChat are available from their Hugging Face repository. Note that the original model and this GGUF version are released under the MIT license.

You can download them using tools like huggingface-cli. However, since the new Aion architecture is not yet supported in the current upstream llama.cpp, it must be built with a dedicated patch for usage. The procedure to apply aion-port.patch included in the repository is as follows:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout bafa0420c1e57edf04d71587d32fe57d45df7b95
curl -LO https://huggingface.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF/resolve/main/aion-port.patch
git apply aion-port.patch
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-server llama-cli

After building, run it with the --jinja option specified, as the chat template is embedded:

./build/bin/llama-server -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:AD-Q4_K_M --jinja -c 32768 -ngl 99

Related Articles

What to Read Next

Sources

This article contains unverified information. We will append an update note once it is confirmed by a primary source.