OrcaSAQ-2-27B Text Generation Model: Our Test Answers, 16GB+ VRAM

September 28, 2026November 11, 2026

At a Glance

Values determined by this site’s code when the information was collected. Dates are JST.

What we checked ourselves

• Ran the publisher’s own code (transformers , , revision 15d20d7e9a) on a cloud GPU on Modal (NVIDIA L4, peak VRAM 14.0GB) and show its answers verbatim.
• It needs about 21% fewer tokens than the Qwen3 tokenizer for the same Japanese text, so about 1.26× as much Japanese fits in the same context length, and generation is faster per character.

Details and conditions are in “Our Own Measurements” below.

Overview

OrcaRouter, an AI gateway operator, has released a quantized version of Qwen3.8-27B compressed to an average of 3.21 bits, named OrcaSAQ2 27B (orcarouter/OrcaSAQ-2-27B), on Hugging Face. The original model, which is 54GB in BF16, has been reduced to a total of 12.3GB across its distribution files. The stated goal of the publisher is to “run a 27B model on a single 16GB GPU," and they present numerical comparisons directly against the BF16 version to show that the difference from the original model is small.

The publisher calls this a “proprietary sensitivity-aware mixed-precision quantization system (SAQ2)" and has not disclosed the details of the method (calibration procedures, bit-width allocation per layer, or packing methods). However, looking at the distribution files’ configuration (quantization_config.json), the quantization format is exl3 (ExLlamaV3 format, version 1.5.1). The publisher’s concurrently released execution kernel repository also describes its contents as “a combination of EXL3 trellis quantization (QTIP family) with mixed-precision allocation determined by search." In other words, it is accurate to read this as not a new format, but rather a clever allocation of bit-widths per layer within the existing EXL3 format.

The base Qwen3.8-27B model itself is covered in Qwen3.8-27B Vision-Language Model: Our Test Answers, 8GB+ VRAM.

Specifications

• Base Model: Qwen3.8-27B (Hybrid architecture of the Qwen3.5 family. Out of 64 layers, 48 are Gated DeltaNet and 16 are standard attention)
• Quantization: EXL3 format, averaging 3.21 bits for the decoder part. Output layer (lm_head) is 6-bit, embeddings are int8, and MTP heads are 4-bit
• Distribution Files: Split into 4 safetensors files, total 12.27GB
• Context Length: 262,144 tokens (design upper limit. The length actually usable on a 16GB GPU is described later)
• Supported Features: Thinking mode, tool calling, speculative decoding via MTP (Multi-Token Prediction)
• Input: Text only. While the base Qwen3.8-27B can read images and videos, this version does not include a visual encoder
• License: Apache-2.0 (inherited from the base Qwen3.8-27B)

Performance

Deviation from the Original Model (BF16)

The publisher provides values comparing the distributed weights themselves using the same evaluation pipeline as the BF16 version, measured on 16,376 tokens of WikiText-2.

The meanings of the three metrics are as follows:

• PPL (Perplexity): Measures how well the model can predict the continuation of text without confusion. Lower is better. The degradation from BF16 is +0.02%, showing almost no difference
• KLD (Kullback-Leibler Divergence): Measures how much the probability distribution of the next token deviates from BF16. Closer to 0 means more faithful to the original model
• Top-1 Match Rate: The proportion of times the “most likely next token chosen" matched the BF16 version

It is important to read these three metrics side by side. Even though the PPL difference is only 0.02%, the Top-1 match rate is 93.2%, meaning roughly once every 15 tokens, a different token is chosen as the most likely candidate compared to BF16. Because PPL is an averaged value across the entire text, individual choices tend to cancel each other out and do not readily show up in the aggregate score. The publisher itself notes in the limitations section that “a 93.2% match rate means some token decisions differ from BF16" and “a +0.02% PPL does not guarantee identical performance on downstream tasks." Rather than summarizing it as “virtually lossless," it is more accurate to understand it as “average predictions barely change, but individual decisions change at a certain rate." Note that comparisons measuring other quantized versions (like GGUF or other EXL3 variants) under the same conditions are not provided, so it is impossible to judge from this data alone whether a KLD of 0.031 is exceptional for the 3-bit range.

Agent Benchmarks (Publisher’s Self-Report)

The model card presents scores of 70.0 on SWE-bench Verified and 58.4 on Terminal-Bench 2.1 in a table alongside published values for Claude Sonnet 4.6, Gemini 3, GPT-5.4, and others. However, this article does not treat these as rankings against other models for the following reasons:

• The publisher itself clarifies that the table is “not a controlled comparison, but published reference values." Agent benchmark scores vary significantly depending on the execution environment (harness), thinking budget, and time limits used
• The specific execution environment and settings used to measure the OrcaSAQ2 scores are not documented
• The most crucial piece of information—"the score of the BF16 version measured under the exact same conditions“—is missing. Therefore, it is impossible to tell from this table how much performance was lost due to quantization

The only meaningful evaluation for the quantized version is the “deviation from BF16" discussed in the previous section. It is best not to read more into the agent benchmark scores than simply that the publisher reported them.

Speed on a 16GB GPU

Values measured with vLLM while restricting GPU memory to 15.7GiB.

Using MTP (speculative decoding) makes generation about 38% faster for single-user use. In exchange, because the draft head takes KV cache from the same memory pool, the available context length is cut roughly in half, and it actually becomes slower when handling many concurrent requests. The choice comes down to using MTP enabled for single-user interactive use, and disabled for batching multiple requests.

The execution kernel repository contains a table with the same intent, but the numbers differ (with a 15.5GiB limit, 1 stream with MTP is 111.2 tok/s, and KV cache without MTP is 59,753 tokens). This is likely due to differences in settings or versions, but both are measurements by the publisher, and independent third-party replication is not yet available.

Strengths and Use Cases

The target use case envisioned by the publisher is agents performing long procedures involving coding, terminal operations, and browser navigation. The publisher’s concern is that even small errors from quantization can cause a single tool call to change, cascading to alter all subsequent states—which is why tasks with long procedures are more susceptible to quantization effects, and why they detailed the “deviation from BF16" so thoroughly.

From a local user’s perspective, this model is suited for those who want to set up a 27B-class model as a server using a 16GB GPU (such as an RTX 5080 or 4080). Since it runs as an OpenAI-compatible API, it can be integrated with coding agents or custom tools.

There are also many points to keep in mind:

• You cannot use a 262K context on 16GB. Because the weights consume about 12.3GB, only about 3 to 4GB remains. The publisher also recommends “starting with a context around 32K on a 16GB GPU and adjusting according to the use case"
• Images cannot be read. The visual capabilities of the base Qwen3.8-27B have been stripped out in this version
• The quantization method is private. You cannot apply the exact same procedure to other models yourself
• The publisher is a company operating an AI gateway, and many models released on Hugging Face are uncensored versions of existing models. This OrcaSAQ2 27B is released as a direct quantization of Qwen3.8-27B rather than an uncensored version

How It Differs from Similar Models

• Difference from general GGUF quantized versions (Q3_K_M, IQ3 families, etc.): While the file sizes are similar, the execution engines are completely different. While GGUF runs out of the box in llama.cpp, Ollama, and LM Studio, this version does not run in them (see the next section). Its strengths lie in providing numerical measurements of deviation from BF16 and the ability to handle multiple concurrent requests using vLLM
• Difference from other EXL3 quantized versions: The EXL3 format itself is the standard format for ExLlamaV3, and other versions around 3 bits could potentially be created. The differences in OrcaSAQ2 lie in searching for layer-wise bit allocation distributions and providing a plugin to run it with vLLM. However, numerical comparisons against other EXL3 versions under identical conditions are not provided
• Difference from the base Qwen3.8-27B (BF16): The required memory drops to under a quarter, but image input capability is lost, and token-level agreement is limited to 93.2%. The choice is between the original model if accuracy is the top priority, or this version if fitting into 16GB is the priority

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 27.8B parameters (taken from the base model Qwen/Qwen3.8-27B)

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

Not usable in Ollama, LM Studio and llama.cpp yet — we have found no GGUF build.

The publisher ships EXL3 / safetensors only. Today it can be run with transformers, using the memory figures in the table above.

License — apache-2.0 (Commercial use allowed): Permits commercial use, modification and redistribution. Redistribution requires including the license and stating changes; includes a patent grant.

Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists. License summaries are not legal advice — check the publisher’s original terms before relying on them.

Our Own Measurements

Values we measured ourselves — not figures copied from the model card. We read this model’s files on our own server (no GPU), and because we cannot run this model on that server, we ran it on a rented cloud GPU (conditions below). How we measure, and the results for every model: Our Measurements.

What We Found

OrcaSAQ-2-27B demonstrates high Japanese token efficiency, allowing more information to be packed into a limited context length. However, it showed instability in knowledge-based questions and long-form generation, such as truncated responses and repeated lines. Its Japanese token efficiency is 546 tokens per 1,000 characters, which is lower than the 688 tokens required by Qwen3, enabling more compact processing of Japanese text.

Testing with the EXL3 3.21bit format on a cloud GPU (NVIDIA L4) showed that the model requires 13.7GB of VRAM for loading, with a peak usage of 14.0GB during generation. The generation speed was 17.19 tokens per second. In this environment, the model successfully followed instructions for tasks such as summarization, haiku creation, and math word problems, correctly identifying the answer of 280 yen for the latter.

Conversely, the model hit the 1,024-token limit and truncated its response when asked to list prefectures without a coastline. During this process, a single line was repeated 11 times, and it only listed 4 out of the 8 correct prefectures while incorrectly including 2 coastal prefectures. Since repeated lines were also observed twice during the drafting of a business email, caution is advised when using this model for long-form generation or specific knowledge retrieval tasks.

Japanese Token Efficiency

It needs about 21% fewer tokens than the Qwen3 tokenizer for the same Japanese text, so about 1.26× as much Japanese fits in the same context length, and generation is faster per character.

Counted with the tokenizer.json of orcarouter/OrcaSAQ-2-27B on a fixed text we wrote ourselves (876 Japanese characters across news, conversation, technical docs, a formal email, travel writing and a recipe) and its English translation. Fewer tokens mean more Japanese fits in the context window.

Running the EXL3 Weights With ExLlamaV3 (Cloud GPU)

llama.cpp does not read the EXL3 format, so we ran this repository’s weights as published with ExLlamaV3, the open-source engine for EXL3, on a rented cloud GPU. We did not run any code from the publisher’s repository, and the run happened in a container with its network cut off.

Conditions: NVIDIA L4 (VRAM 22GB) on Modal, ExLlamaV3 1.5.3, PyTorch 2.8.0+cu128, orcarouter/OrcaSAQ-2-27B at revision 15d20d7e9a (EXL3 3.21bit, greedy decoding, thinking off), measured 2026-10-01 (JST).

The speed is a reference for this one GPU and engine; it is not a speed you would get with llama.cpp or on a different GPU. The embedding layer in this repository is stored as int8 values plus a per-row scale, which ExLlamaV3 cannot read as is, so before loading we multiplied them back into a bf16 weight ourselves (the code is in the procedure below). The publisher has not documented this conversion, so this is our own reading of the format.

Answers to Our Japanese Questions (Verbatim)

We ask every model the same five questions in Japanese under the conditions above (temperature 0, thinking turned off where the chat template allows it, up to 1024 tokens) and publish the answers unedited. We do not grade them; where a question has a single correct answer, we show it for comparison.

Summarization

次の記事を3文で要約してください。

みどり市は今年の春、市内に三つある公立図書館のうち最も古い中央図書館の改修を終え、十月一日に再開館した。改修は約一年半にわたり、総事業費は約十二億円。このうち約四億円は国の補助金でまかなわれた。

最大の変化は、閲覧席の配置である。以前は館内の座席の大半が学習用の机で、試験期間になると中高生で埋まり、高齢の利用者や小さな子ども連れが座れないという苦情が毎年寄せられていた。改修後は座席数を百二十席から二百席に増やしたうえで、学習用の机を三階にまとめ、一階は会話ができる「にぎわいフロア」、二階は静かに本を読む「閲覧フロア」に分けた。一階には予約制のグループ学習室が四室と、飲み物を持ち込めるカフェスペースも設けられた。

蔵書の面でも見直しが進んだ。貸出の少ない古い資料約三万冊を市内の他の館や保存書庫に移し、代わりに子ども向けの絵本と、介護・子育て・仕事探しといった暮らしに関わる実用書を約一万五千冊増やした。電子書籍の貸出サービスも始まり、利用登録をすれば自宅から一度に三冊まで、二週間借りられる。

建物そのものにも手が入った。築四十年を超えていた空調設備を入れ替え、屋上には太陽光パネルを設置して、館内で使う電力の約二割をまかなえるようにした。車いすの利用者から要望の多かったエレベーターは一基から二基に増え、書架の間の通路も以前より三十センチ広げられた。入口には段差のないスロープが新たに設けられ、授乳室と多目的トイレも各階に一つずつ置かれた。

一方で課題も残る。駐車場は改修前と同じ四十台のままで、週末には近くの道路に順番待ちの車が並んだ。市は隣接する市有地を臨時駐車場として開放したが、恒久的な対策は決まっていない。また、開館時間を平日の夜九時まで延ばしたことで職員の負担が増えており、職員団体は人員の追加を求めている。

利用者の受け止めはおおむね好意的だ。週に二度、孫を連れて来るという七十代の女性は「前は子どもが声を出すと周りの目が気になったが、一階なら気兼ねなく絵本を読んであげられる」と話す。一方、受験を控えた高校生の中には「学習席が三階だけになり、夕方はすぐに満席になる」と不満を漏らす人もいる。図書館は、学習席の空き状況をホームページで確認できる仕組みを年内に導入する予定だ。

再開館に合わせて、催しも増えた。毎週土曜の午前には読み聞かせ会、月に一度は地元の農家や商店主を講師に招く「まちの先生」講座が開かれる。十一月には、市内の中学生が選んだ本を紹介する展示も予定されている。

館長の山本和子さんは「本を借りる場所から、人が集まって過ごす場所へ変えたかった。再開館からの一か月で来館者は以前の一・八倍になった」と話す。市は来年度、残る二つの図書館についても利用者への聞き取りを行い、改修の必要性を検討する方針だ。

(Summarize the following article in three sentences. (The article, about 1,200 Japanese characters, describes the renovation and reopening of a city’s central library: the budget, the new floor layout and seating, changes to the collection and e-book lending, building and accessibility work, remaining issues such as parking and staff workload, users’ reactions, new events, and the city’s next steps.))

みどり市中央図書館は約12億円の改修を経て再開し、フロアごとの機能分化や蔵書の見直し、バリアフリー対応などを通じて、単なる閲覧施設から地域交流の場へと転換した。その結果、来館者は以前比1.8倍に増加し、多様な利用者のニーズに応える環境が整った一方、駐車場不足や職員負担増といった課題も残っている。市は今後の利用者調査を通じて残る2館の改修必要性を検討し、図書館機能のさらなる充実を図る方針を示している。

Formal business email

取引先の担当者に、来週火曜日の打ち合わせを水曜日に変更してほしいとお願いするメールを、丁寧な敬語で書いてください。

(Write an email in polite Japanese (keigo) asking a client’s contact person to move next Tuesday’s meeting to Wednesday.)

件名：【ご相談】来週火曜日の打ち合わせ日程変更のお願い

○○株式会社
○○部 ○○様

いつも大変お世話になっております。
株式会社△△の△△です。

突然のご連絡となり、大変失礼いたします。

来週火曜日（○月○日）に予定しております打ち合わせの件で、誠に恐れ入りますが、日程の変更をお願いしたくご連絡いたしました。

社内事情により、当日の参加予定者が都合つかない状況となりましたため、もし可能であれば、**翌日の水曜日（○月○日）**に変更させていただくことは可能でしょうか。

つきましては、以下の日程で改めてご調整いただけますと幸いです。

■希望日程
○月○日（水） 10:00〜12:00
※上記の時間帯で調整が難しい場合は、水曜日の他の時間帯や、以降の日程でもご相談に応じます。

ご多忙のところ、急なお願いとなり大変申し訳ございません。
ご確認のほど、何卒よろしくお願い申し上げます。

―――――――――――――――――――
株式会社△△
△△部 △△（氏名）
TEL: 03-XXXX-XXXX
Mail: xxxxx@xxxx.com
―――――――――――――――――――

Word problem

りんごが1個120円、みかんが1個45円です。りんごを3個とみかんを8個買って、1,000円札で払いました。おつりはいくらですか。考え方も示してください。

(Apples cost 120 yen each and mandarins cost 45 yen each. You buy 3 apples and 8 mandarins and pay with a 1,000-yen bill. How much change do you get? Show your reasoning.)

おつりの金額を求めるために、まずりんごとみかんの合計代金を計算し、そこから1,000円を引く手順で考えます。

### 1. 考え方の手順

1.  **りんごの代金**を計算する。
    *   単価 × 個数
2.  **みかんの代金**を計算する。
    *   単価 × 個数
3.  **合計代金**を計算する。
    *   りんごの代金 + みかんの代金
4.  **おつり**を計算する。
    *   支払った金額（1,000円） - 合計代金

---

### 2. 計算過程

**① りんごの代金**
$$ 120 \text{円} \times 3 = 360 \text{円} $$

**② みかんの代金**
$$ 45 \text{円} \times 8 = 360 \text{円} $$

**③ 合計代金**
$$ 360 \text{円} + 360 \text{円} = 720 \text{円} $$

**④ おつり**
$$ 1,000 \text{円} - 720 \text{円} = 280 \text{円} $$

---

### 3. 答え

おつりは **280円** です。

Reference: The correct answer is 280 yen (120×3 = 360, 45×8 = 360, 1,000 − 720 = 280).

Knowledge

日本の都道府県のうち、海に面していない県をすべて挙げてください。

(List all of Japan’s prefectures that do not border the sea.)

日本の都道府県の中で、海に面していない（内陸県）のは以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **三重県**（※注：三重県は伊勢湾に面しているため、厳密には海に面しています。しかし、多くの文脈で「内陸県」として扱われることがありますが、地理的には海に面しています。ここでは**海に面していない**という厳密な定義に基づき、三重県は除外します。）

※訂正：三重県は伊勢湾に面しているため、海に面しています。したがって、内陸県ではありません。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**（琵琶湖は湖であり、海ではないため内陸県とされます）
5. **愛知県**（※注：愛知県は太平洋と伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：愛知県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **三重県**（※伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：三重県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **愛知県**（※太平洋と伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：愛知県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **三重県**（※伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：三重県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **愛知県**（※太平洋と伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：愛知県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **三重県**（※伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：三重県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **愛知県**（※太平洋と伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：愛知県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **三重県**（※伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：三重県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県**
3. **岐阜県**
4. **滋賀県**
5. **愛知県**（※太平洋と伊勢湾に面しているため、海に面しています。内陸県ではありません。）

※再度確認：愛知県は海に面しています。

正しい内陸県（海に面していない県）は以下の**9県**です。

1. **山梨県**
2. **長野県

(Cut off at the 1024-token limit.)

Reference: The correct answer is eight prefectures: Tochigi, Gunma, Saitama, Yamanashi, Nagano, Gifu, Shiga and Nara.

Constrained writing

「春」をテーマにした俳句を3つ、五・七・五の音数を守って作ってください。それぞれに一言の説明を付けてください。

(Write three haiku on the theme of spring, keeping the 5-7-5 syllable pattern, each with a one-line explanation.)

ご指定の条件に基づき、「春」をテーマにした俳句を3つ作成しました。それぞれに一言の解説を添えています。

1. **花吹雪 舞い散る空に 春の風**
   > 解説：桜の花びらが舞い散る情景と、その中を抜ける柔らかな風で、春の移ろいを表現しました。

2. **新緑の 木漏れ日揺れて 春の光**
   > 解説：若葉が茂り始めた木々から差し込む、揺れる光の美しさを捉えました。

3. **蛙鳴く 田んぼの春の 朝の空気**
   > 解説：田んぼで鳴き始める蛙の声と、清々しい朝の空気感で、春の生命力を表現しました。

Setup, Download and Run Steps (What We Actually Ran)

We split the work into two containers on Modal. The downloading container is a Debian (slim) image with Python 3.12; it only fetches files. The running container is nvidia/cuda:12.8.1-devel-ubuntu22.04 with Python 3.12 and has no network access. ExLlamaV3’s CUDA extension is compiled while the image is built, so no compiling happens on the GPU. These are the commands that built the two environments.

# 1. Environment for downloading (no GPU)
python -m pip install huggingface_hub

# 2. Environment for running (GPU, network blocked)
export TORCH_CUDA_ARCH_LIST="8.0;8.9" MAX_JOBS="4"
pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install transformers jinja2 ninja setuptools wheel
pip install --no-build-isolation exllamav3==1.5.3

The first container ran download(job) from remote_code_remote.py and the second ran run(job) from exl3_remote.py (both written by us; the second in full below). Files go under /models/<dir>:

"""
Modal のコンテナの中で動く、ExLlamaV3 による EXL3 形式のモデルの実測(2026-09-30)。
lmw/lab/gpu_modal.py の run_exl3 から呼ばれる。ダウンロードは remote_code_remote.download を使う
(重みは safetensors だけ・配布元のコードは落とさず実行もしない)。

実行するコンテナはネットワーク遮断・Modal の機能も使えない・Volume は読み取り専用。ExLlamaV3 は pip の公開
パッケージで、配布元のリポジトリのコードは入れない(配布元が別に配っているカーネルも使わない)。
記事執筆用の LLM は呼ばない。回答の良し悪しも判定しない(原文を返すだけ)。
"""
import json
import os
import platform
import time
from datetime import datetime, timezone

from lmw.lab.remote_code_remote import MOUNT, _log, installed_packages, plain, split_thinking

from lmw.config import (
    LAB_EXL3_CUDA_ARCHS, LAB_EXL3_CUDA_IMAGE, LAB_EXL3_PYTHON, LAB_EXL3_TORCH, LAB_EXL3_TORCH_INDEX, LAB_EXL3_VERSION,
)

# 実行用のイメージの作り方(lmw/lab/gpu_modal.py が実行し、記事の手順にも同じ値を載せる)
BASE_IMAGE = LAB_EXL3_CUDA_IMAGE
PYTHON_VERSION = LAB_EXL3_PYTHON
BUILD_ENV = {"TORCH_CUDA_ARCH_LIST": LAB_EXL3_CUDA_ARCHS, "MAX_JOBS": "4"}
INSTALL_COMMANDS = (
    f"pip install torch=={LAB_EXL3_TORCH} --index-url {LAB_EXL3_TORCH_INDEX}",
    "pip install transformers jinja2 ninja setuptools wheel",
    f"pip install --no-build-isolation exllamav3=={LAB_EXL3_VERSION}",
)

def _stop_ids(path: str, tokenizer) -> list[int]:
    """generation_config.json の eos(整数か一覧)。無ければトークナイザの eos。"""
    ids: list[int] = []
    try:
        with open(os.path.join(path, "generation_config.json"), encoding="utf-8") as f:
            eos = json.load(f).get("eos_token_id")
        ids = [int(x) for x in (eos if isinstance(eos, list) else [eos])] if eos is not None else []
    except (OSError, ValueError):
        pass
    if not ids and tokenizer.eos_token_id is not None:
        ids = [int(tokenizer.eos_token_id)]
    return ids

def _prompt_text(path: str, text: str) -> str:
    from transformers import AutoTokenizer
    tok = AutoTokenizer.from_pretrained(path, local_files_only=True)
    if not getattr(tok, "chat_template", None):
        return ""
    messages = [{"role": "user", "content": text}]
    try:
        return tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    except TypeError:
        return tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

def _dequantize_embedding(src: str) -> tuple[str, str]:
    """
    埋め込みが qweight(int8)+scales(行ごとの bf16)で保存されている配布(quantization_config.json は
    bf16 の weight と書いている)を、ExLlamaV3 が読める weight に直した作業用フォルダを /tmp に作る
    (Volume は読み取り専用)。戻り値: (フォルダ, 直したことの説明)。埋め込みが標準の形なら何もしない。
    """
    import glob
    import shutil
    import torch
    from safetensors import safe_open
    from safetensors.torch import save_file
    prefix = "model.language_model.embed_tokens"
    holder = None
    for f in sorted(glob.glob(os.path.join(src, "*.safetensors"))):
        with safe_open(f, "pt") as st:
            keys = set(st.keys())
        if f"{prefix}.weight" in keys:
            return src, ""
        if f"{prefix}.qweight" in keys:
            holder = f
    if holder is None:
        return src, ""
    with safe_open(holder, "pt") as st:
        q = st.get_tensor(f"{prefix}.qweight")
        scales = st.get_tensor(f"{prefix}.scales")
    weight = (q.to(torch.float32) * scales.to(torch.float32)[:, None]).to(torch.bfloat16).contiguous()
    work = "/tmp/exl3-model"
    shutil.rmtree(work, ignore_errors=True)
    os.makedirs(work)
    for name in os.listdir(src):
        os.symlink(os.path.join(src, name), os.path.join(work, name))
    save_file({f"{prefix}.weight": weight}, os.path.join(work, "embed-dequantized.safetensors"))
    return work, "埋め込み: int8 の qweight × 行ごとの scales を bf16 の weight に直して読み込み"

def run(job: dict) -> dict:
    """
    job: {"dir", "prompts": [{"id", "ja"}], "max_tokens", "cache_tokens", "answer_budget_sec"}。
    チャットテンプレートが無ければ {"no_chat_template": True}。
    """
    import exllamav3
    import torch
    from exllamav3 import Cache, Config, Generator, Job, Model, Tokenizer
    from exllamav3.generator.sampler import CustomSampler, SS_Argmax
    os.environ["HF_HUB_OFFLINE"] = "1"
    os.environ["TRANSFORMERS_OFFLINE"] = "1"
    t0 = time.time()
    path = os.path.join(MOUNT, job["dir"])
    gpu = torch.cuda.get_device_name(0)
    vram_total = torch.cuda.get_device_properties(0).total_memory
    from importlib import metadata
    version = getattr(exllamav3, "__version__", "") or metadata.version("exllamav3")
    _log(t0, f"GPU: {gpu}・exllamav3 {version}・torch {torch.__version__}")
    with open(os.path.join(path, "quantization_config.json"), encoding="utf-8") as f:
        quant = json.load(f)
    record = {"runtime": "exllamav3", "gpu": gpu, "vram_total": int(vram_total), "exllamav3": str(version),
              "torch": str(torch.__version__), "packages": installed_packages(),
              "python": platform.python_version(), "cache_tokens": int(job["cache_tokens"]),
              "quant_method": quant.get("quant_method"), "quant_version": quant.get("version"),
              "bits": quant.get("bits"), "head_bits": quant.get("head_bits"),
              "measured_at": datetime.now(timezone.utc).isoformat()}
    first = _prompt_text(path, job["prompts"][0]["ja"])
    if not first:
        _log(t0, "チャットテンプレートがありません")
        return plain({**record, "no_chat_template": True})
    path, note = _dequantize_embedding(path)
    if note:
        record["embed_dequant"] = note
        _log(t0, note)
    config = Config.from_directory(path)
    model = Model.from_config(config)
    cache = Cache(model, max_num_tokens=job["cache_tokens"])
    model.load(progressbar=False)
    tokenizer = Tokenizer.from_config(config)
    generator = Generator(model=model, cache=cache, tokenizer=tokenizer)
    torch.cuda.synchronize()
    record["vram_loaded"] = int(torch.cuda.memory_allocated())
    record["load_sec"] = round(time.time() - t0)
    _log(t0, f"読み込み: VRAM {record['vram_loaded'] / 1024 ** 3:.1f}GB・{record['load_sec']}秒")
    torch.cuda.reset_peak_memory_stats()
    stops = _stop_ids(path, tokenizer)
    sampler = CustomSampler([SS_Argmax()])
    outputs = []
    stop_at = t0 + job.get("answer_budget_sec", 10 ** 9)
    for prompt in job["prompts"]:
        slowest = min((o["gen_tps"] for o in outputs if o.get("gen_tps")), default=None)
        worst = job["max_tokens"] / slowest if slowest else 0
        if time.time() + worst > stop_at:
            _log(t0, f"時間の上限が近いため、残りの問い({prompt['id']} から)は尋ねません")
            record["unanswered"] = [p["id"] for p in job["prompts"][len(outputs):]]
            break
        text = _prompt_text(path, prompt["ja"])
        ids = tokenizer.encode(text, add_bos=False, encode_special_tokens=True)
        started = time.time()
        gen_job = Job(input_ids=ids, max_new_tokens=job["max_tokens"], stop_conditions=stops, sampler=sampler,
                      decode_special_tokens=True)
        generator.enqueue(gen_job)
        pieces: list[str] = []
        count = 0
        eos_reason = ""
        while generator.num_remaining_jobs():
            for result in generator.iterate():
                if result.get("stage") != "streaming":
                    continue
                pieces.append(result.get("text", ""))
                if result.get("eos"):
                    count = int(result.get("new_tokens", 0))
                    eos_reason = str(result.get("eos_reason", ""))
        torch.cuda.synchronize()
        seconds = time.time() - started
        answer, thinking = split_thinking("".join(pieces), text[-64:])
        for special in ("<|im_end|>", "<|endoftext|>", "<|im_start|>"):
            answer = answer.replace(special, "")
        outputs.append({
            "id": prompt["id"],
            "content": answer.strip(),
            "reasoning_chars": len(thinking.strip()),
            "finish_reason": "length" if eos_reason == "max_new_tokens" or count >= job["max_tokens"] else "stop",
            "completion_tokens": count,
            "gen_tps": round(count / seconds, 2) if seconds > 0 and count else None,
            "prompt_tokens": int(ids.shape[-1]),
        })
        _log(t0, f"回答 {prompt['id']}: {count}トークン・{outputs[-1]['gen_tps']} トークン/秒")
    record["vram_peak"] = int(torch.cuda.max_memory_allocated())
    record["outputs"] = outputs
    gen = sorted(o["gen_tps"] for o in outputs if o.get("gen_tps"))
    if gen:
        record["tg_tps"] = gen[len(gen) // 2]
    return plain(record)

The job passed to download (as used):

{
  "repo": "orcarouter/OrcaSAQ-2-27B",
  "revision": "15d20d7e9ae4fd89d1a47878f69381760169445b",
  "files": [
    "chat_template.jinja",
    "config.json",
    "generation_config.json",
    "merges.txt",
    "model-00001-of-00004.safetensors",
    "model-00002-of-00004.safetensors",
    "model-00003-of-00004.safetensors",
    "model-00004-of-00004.safetensors",
    "model.safetensors.index.json",
    "preprocessor_config.json",
    "quantization_config.json",
    "tokenizer.json",
    "tokenizer_config.json",
    "video_preprocessor_config.json",
    "vocab.json"
  ],
  "dir": "run-1790831352"
}

The job passed to run (as used):

{
  "dir": "run-1790831352",
  "max_tokens": 1024,
  "cache_tokens": 16384,
  "answer_budget_sec": 2220,
  "prompts": [
    {
      "id": "summary",
      "ja": "次の記事を3文で要約してください。\n\nみどり市は今年の春、市内に三つある公立図書館のうち最も古い中央図書館の改修を終え、十月一日に再開館した。改修は約一年半にわたり、総事業費は約十二億円。このうち約四億円は国の補助金でまかなわれた。\n\n最大の変化は、閲覧席の配置である。以前は館内の座席の大半が学習用の机で、試験期間になると中高生で埋まり、高齢の利用者や小さな子ども連れが座れないという苦情が毎年寄せられていた。改修後は座席数を百二十席から二百席に増やしたうえで、学習用の机を三階にまとめ、一階は会話ができる「にぎわいフロア」、二階は静かに本を読む「閲覧フロア」に分けた。一階には予約制のグループ学習室が四室と、飲み物を持ち込めるカフェスペースも設けられた。\n\n蔵書の面でも見直しが進んだ。貸出の少ない古い資料約三万冊を市内の他の館や保存書庫に移し、代わりに子ども向けの絵本と、介護・子育て・仕事探しといった暮らしに関わる実用書を約一万五千冊増やした。電子書籍の貸出サービスも始まり、利用登録をすれば自宅から一度に三冊まで、二週間借りられる。\n\n建物そのものにも手が入った。築四十年を超えていた空調設備を入れ替え、屋上には太陽光パネルを設置して、館内で使う電力の約二割をまかなえるようにした。車いすの利用者から要望の多かったエレベーターは一基から二基に増え、書架の間の通路も以前より三十センチ広げられた。入口には段差のないスロープが新たに設けられ、授乳室と多目的トイレも各階に一つずつ置かれた。\n\n一方で課題も残る。駐車場は改修前と同じ四十台のままで、週末には近くの道路に順番待ちの車が並んだ。市は隣接する市有地を臨時駐車場として開放したが、恒久的な対策は決まっていない。また、開館時間を平日の夜九時まで延ばしたことで職員の負担が増えており、職員団体は人員の追加を求めている。\n\n利用者の受け止めはおおむね好意的だ。週に二度、孫を連れて来るという七十代の女性は「前は子どもが声を出すと周りの目が気になったが、一階なら気兼ねなく絵本を読んであげられる」と話す。一方、受験を控えた高校生の中には「学習席が三階だけになり、夕方はすぐに満席になる」と不満を漏らす人もいる。図書館は、学習席の空き状況をホームページで確認できる仕組みを年内に導入する予定だ。\n\n再開館に合わせて、催しも増えた。毎週土曜の午前には読み聞かせ会、月に一度は地元の農家や商店主を講師に招く「まちの先生」講座が開かれる。十一月には、市内の中学生が選んだ本を紹介する展示も予定されている。\n\n館長の山本和子さんは「本を借りる場所から、人が集まって過ごす場所へ変えたかった。再開館からの一か月で来館者は以前の一・八倍になった」と話す。市は来年度、残る二つの図書館についても利用者への聞き取りを行い、改修の必要性を検討する方針だ。"
    },
    {
      "id": "keigo",
      "ja": "取引先の担当者に、来週火曜日の打ち合わせを水曜日に変更してほしいとお願いするメールを、丁寧な敬語で書いてください。"
    },
    {
      "id": "arithmetic",
      "ja": "りんごが1個120円、みかんが1個45円です。りんごを3個とみかんを8個買って、1,000円札で払いました。おつりはいくらですか。考え方も示してください。"
    },
    {
      "id": "prefectures",
      "ja": "日本の都道府県のうち、海に面していない県をすべて挙げてください。"
    },
    {
      "id": "haiku",
      "ja": "「春」をテーマにした俳句を3つ、五・七・五の音数を守って作ってください。それぞれに一言の説明を付けてください。"
    }
  ]
}

aiohappyeyeballs==2.6.1 aiohttp==3.12.7 aiosignal==1.3.2 annotated-doc==0.0.5 annotated-types==0.8.0 anyio==4.15.1 attrs==25.3.0 cbor2==5.7.0 certifi==2026.7.22 click==8.5.0 exllamav3==1.5.3 filelock==3.32.3 frozenlist==1.6.0 fsspec==2026.7.0 grpclib==0.4.8 h11==0.16.0 h2==4.2.0 hf-xet==1.6.0 hpack==4.1.0 httpcore==1.0.9 httpx==0.28.1 huggingface_hub==1.33.0 hyperframe==6.1.0 idna==3.20 Jinja2==3.1.6 llguidance==1.9.0 marisa-trie==1.4.1 markdown-it-py==4.2.0 MarkupSafe==3.0.3 mdurl==0.1.2 mpmath==1.3.0 multidict==6.4.4 networkx==3.6.1 ninja==1.13.2 numpy==2.5.3 nvidia-cublas-cu12==12.8.4.1 nvidia-cuda-cupti-cu12==12.8.90 nvidia-cuda-nvrtc-cu12==12.8.93 nvidia-cuda-runtime-cu12==12.8.90 nvidia-cudnn-cu12==9.10.2.21 nvidia-cufft-cu12==11.3.3.83 nvidia-cufile-cu12==1.13.1.3 nvidia-curand-cu12==10.3.9.90 nvidia-cusolver-cu12==11.7.3.90 nvidia-cusparse-cu12==12.5.8.93 nvidia-cusparselt-cu12==0.7.1 nvidia-nccl-cu12==2.27.3 nvidia-nvjitlink-cu12==12.8.93 nvidia-nvtx-cu12==12.8.90 packaging==26.3 pillow==12.3.0 pip==23.3.2 propcache==0.3.1 protobuf==6.31.1 pydantic==2.13.5 pydantic_core==2.46.5 Pygments==2.21.0 PyYAML==6.0.3 regex==2026.9.29 rich==15.0.0 safetensors==0.8.0 setuptools==69.0.3 shellingham==1.5.4 sympy==1.14.0 tokenizers==0.23.2 torch==2.8.0+cu128 tqdm==4.70.1 transformers==5.17.0 triton==3.4.0 typer==0.27.2 typing-inspection==0.4.4 typing_extensions==4.16.0 wheel==0.48.0 yarl==1.20.0

aiohappyeyeballs==2.6.1 aiohttp==3.12.7 aiosignal==1.3.2 anyio==4.15.1 attrs==25.3.0 cbor2==5.7.0 certifi==2025.4.26 click==8.5.0 filelock==4.0.7 frozenlist==1.6.0 fsspec==2026.9.0 grpclib==0.4.8 h11==0.16.0 h2==4.2.0 hf-xet==1.6.0 hpack==4.1.0 httpcore2==2.13.1 httpx2==2.13.1 huggingface_hub==2.0.0 hyperframe==6.1.0 idna==3.20 multidict==6.4.4 packaging==26.3 pip==25.1.1 propcache==0.3.1 protobuf==6.31.1 PyYAML==6.0.3 tqdm==4.70.1 truststore==0.10.4 typing_extensions==4.16.0 uv==0.7.19 wheel==0.45.1 yarl==1.20.0

Recent Models in the Same Size Class

Models with 15–40B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.

How to Get It

• Distribution: Hugging Face orcarouter/OrcaSAQ-2-27B (safetensors, EXL3 format)
• Does not run in llama.cpp, Ollama, or LM Studio. The execution kernel repository explicitly states: “Since llama.cpp can only read its own quantization codebooks, these weights cannot be used. Converting them to GGUF would require re-quantizing with a different quantizer"
• When running with vLLM: In addition to vLLM, you need to install the publisher’s plugin (Continuum-AI-Corp/OrcaSAQ2-kernel). The model card suggests pip install git+https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel, and the kernel repository also provides startup configurations and Docker images tailored for 16GB. To correctly receive thinking outputs and tool calls, append --reasoning-parser qwen3 and --tool-call-parser qwen3_xml
• When running with ExLlamaV3: The kernel repository notes that “for single-user use on consumer GPUs, it is lighter than vLLM and better suited." However, you must first apply a patch to read embeddings packed in int8. Note that using MTP in ExLlamaV3 allocates separate caching for draft heads, increasing memory usage by about 10GB, which means it will not fit on a 16GB GPU
• Recommended sampling settings: temperature 1.0, top_p 0.95, top_k 20. Thinking mode is enabled by default
• The execution kernel repository was just created on September 24, making it very recent since release. It is wise to anticipate potential issues where it may not run smoothly depending on your environment
• License: Model is Apache-2.0 (inherited from the base Qwen3.8-27B). Please review the original license text before use

Related Articles

• Hemmingway-1 Text Generation Model: 12GB+ VRAM, GGUF Builds
• Signal-3.8-27B-GGUF: Our Test Answers, 16GB+ VRAM
• Qwen3.8-27B Vision-Language Model: Our Test Answers, 8GB+ VRAM
• clef Structured Decision-Making Model: 12GB+ VRAM, GGUF Builds

What to Read Next

• Find models by VRAM (This model runs from the 16GB tier) → Other models that run on a 16GB GPU
• Explore the same model family → Qwen3.8 family overview (8 articles, 17 converted builds)
• What EXL3 mean and where to get this model → EXL2 / EXL3 format guide and models / Quantization and model-format glossary
• Learn about the publisher → Alibaba (Qwen): models, licenses and articles
• Other models for the same task → Other text generation models

Sources

• https://huggingface.co/orcarouter/OrcaSAQ-2-27B
• https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel
• https://huggingface.co/Qwen/Qwen3.8-27B

Update History

• 2026-09-28: Added our own measurements: Japanese token efficiency.
• 2026-09-29: Verified the content against the official primary source.
• 2026-10-01: Added our own measurements: VRAM and answers to Japanese questions with the publisher’s own code.
• 2026-10-01: Changed the title to show what the article covers (VRAM requirements, file list, etc.).