Aion-1.0-Instruct-2.3B-GGUF 2.3B Text Generation Model: 4GB+ VRAM

At a Glance
| Item | Value |
|---|---|
| Repository | AtomicChat/Aion-1.0-Instruct-2.3B-GGUF |
| Publisher guide | Microsoft: models and licenses |
| Published | 2026-11-19 |
| License | mit |
| Formats | GGUF |
| Source type | Unverified (not confirmed by a primary source) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
It is reported that the GGUF version of “Aion 1.0 Instruct 2.3B," which is said to be Microsoft’s next-generation Windows on-device SLM (Small Language Model), has been released by AtomicChat. Note that this information has not been officially confirmed. This model is stated to be a public release aimed at introduction and ensuring transparency, ahead of the scheduled rollout of an optimized version as a Windows OS SLM by the end of January 2027.
Specifications
- Parameters: 2.29B (including 0.41B for tied embedding / head)
- Layers: 28 (21 sliding-window layers (2048) + 7 full attention layers)
- Attention: GQA, 24 query heads, 8 KV heads, head dim 128, QK RMSNorm
- Hidden / FFN: 2048 / 8192 (SwiGLU)
- RoPE: theta 32000; YaRN x16 from 2048 on the full-attention layers
- Context length: 32,768 tokens
- Vocabulary: 200,029 (o200k)
Performance
According to the publisher’s measurements, the performance of the original model microsoft/Aion-1.0-Instruct-2.3B is as follows:
| Category | Benchmark | Score |
|---|---|---|
| Knowledge | MMLU | 55.15 |
| Knowledge | MMLU-Pro | 33.35 |
| Knowledge | MMMLU | 45.51 |
| Knowledge | GPQA | 28.18 |
| Math | GSM8K | 74.91 |
| Math | GSMPlus | 49.56 |
| Math | MATH-500 | 48.00 |
| Instruction following | IFEval | 83.90 |
| Instruction following | IFBench | 32.00 |
| Instruction following | Multi-IF | 64.70 |
| Translation | chrF++ | 43.18 |
From this table, it can be seen that the original model shows high scores, such as 74.91 on GSM8K, which measures the ability to solve math word problems, and 83.90 on IFEval, which measures the ability to follow format instructions. On the other hand, it scores 28.18 on GPQA, which handles difficult scientific questions, a relatively low figure compared to other metrics. Overall, this suggests that the model is designed to deliver high performance in specific tasks despite being lightweight.
Additionally, regarding the quantization precision of the released GGUF version, the following comparison table is provided:
| file | mean KLD | same top token |
|---|---|---|
BF16 |
0 | 100% |
Q8_0 |
0.0029 | 96.6% |
AD-Q5_K_M |
0.0062 | 95.1% |
AD-Q4_K_M |
0.0131 | 93.0% |
AD-IQ3_S |
0.0705 | 84.1% |
According to this table, AD-Q4_K_M, recommended by AtomicChat, maintains a “same top token" ratio (the proportion of selecting the same next token as the original model) of 93.0% compared to the original BF16, achieving weight reduction while suppressing degradation due to quantization. KLD (Kullback-Leibler divergence), an indicator of how far the quantized distribution deviates from the original distribution, is kept relatively low at 0.0131 for AD-Q4_K_M. For the most lightweight AD-IQ3_S, the probability of selecting the same token as the original model drops to 84.1%.
Strengths and Use Cases
The original model microsoft/Aion-1.0-Instruct-2.3B is primarily trained for general-purpose text scenarios such as summarization, rewriting, document question answering, and chatting. It supports 8 languages: English, German, Spanish, Portuguese, Italian, French, Chinese, and Japanese. Furthermore, equipped with a 32K context length, it is capable of processing multi-page documents or long email threads locally. However, the original model’s model card points out that it is not specifically trained for world knowledge and has limitations in advanced multi-step reasoning.
Hardware Requirements
Estimated requirements (calculated by Local Model Watch) — 2.3B parameters (taken from the base model microsoft/Aion-1.0-Instruct-2.3B)
| Your VRAM | Quantization | File size | Est. memory needed |
|---|---|---|---|
| 4GB (laptop iGPU / phone class) | Q8_0 | 2.3GB | 2.7GB |
| 8GB (RTX 4060 / 3060 Ti, etc.) | BF16 | 4.3GB | 5.1GB |
Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.
Can You Run It Locally?
Runs in Ollama, LM Studio and llama.cpp as-is.
It is distributed in GGUF, so no conversion is needed.
License — mit (Commercial use allowed): Permits commercial use, modification and redistribution, provided the copyright notice and license text are retained.
Compression: the Q8_0 build measures 8.63 bits per weight — about 54% the size of the original 16-bit weights, calculated by this site from the actual file sizes.
Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists. License summaries are not legal advice — check the publisher’s original terms before relying on them.
Recent Models in the Same Size Class
Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.
→ Scroll horizontally to see all columns
| Model | Parameters | Smallest VRAM tier | License | Our article |
|---|---|---|---|---|
| ggml-org/LFM3-1.7B-GGUF | 1.7B | 4GB | lfm1.0 | LFM3-1.7B-GGUF Text Generation Model: Our Test Answers, 4GB+ VRAM (2026-11-18) |
| openbmb/MiniCPM-V-4.7-35B-A3B-DSpark-gguf | 389M | 4GB | apache-2.0 | MiniCPM-V-4.7-35B-A3B-DSpark-gguf Draft Model for Speedup: 4GB+ VRAM (2026-11-17) |
| openbmb/MiniCPM-V-4.7-1B-DSpark-gguf | 214M | 4GB | apache-2.0 | MiniCPM-V-4.7-1B-DSpark-gguf Text Generation Model: 4GB+ VRAM (2026-11-17) |
| IFM/Jais-3-0.9B-GGUF | 1.1B | 4GB | apache-2.0 | Jais-3-0.9B-GGUF Bilingual Dense Model: Our Test Answers, 4GB+ VRAM (2026-11-17) |
| nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 | 4.0B | 4GB | openmdw-1.1 | NVIDIA Releases Dory1.0-4B: A Recurrent 4B LLM with Dynamic Loops (2026-11-17) |
How to Get It
The GGUF version files released by AtomicChat are available from their Hugging Face repository. Note that the original model and this GGUF version are released under the MIT license.
You can download them using tools like huggingface-cli. However, since the new Aion architecture is not yet supported in the current upstream llama.cpp, it must be built with a dedicated patch for usage. The procedure to apply aion-port.patch included in the repository is as follows:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout bafa0420c1e57edf04d71587d32fe57d45df7b95
curl -LO https://huggingface.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF/resolve/main/aion-port.patch
git apply aion-port.patch
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-server llama-cli
After building, run it with the --jinja option specified, as the chat template is embedded:
./build/bin/llama-server -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:AD-Q4_K_M --jinja -c 32768 -ngl 99
Related Articles
- Rumble-2-Flash-GGUF 45-Language ASR Model: 4GB+ VRAM, File List
- Maxwell-V1-Flash-GGUF 35B MoE Model: 24GB+ VRAM
What to Read Next
- Find models by VRAM (This model runs from the 4GB tier) → Other models that run on a 8GB GPU
- Engines that run this model → llama.cpp / Ollama
- What Q8_0 mean and where to get this model → GGUF format guide and models / Quantization and model-format glossary
- Learn about the publisher → Microsoft: models, licenses and articles
- Other models for the same task → Other text generation models
Sources
- AtomicChat/Aion-1.0-Instruct-2.3B-GGUF
- microsoft/Aion-1.0-Instruct-2.3B
- AtomicChat/Aion-1.0-Instruct-2.3B-GGUF-metrics
This article contains unverified information. We will append an update note once it is confirmed by a primary source.