Fast Structured Generation on Apple Silicon with MLX and Qwen

September 17, 2026September 18, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

An implementation and model for the MLX framework, harshatheg/Qwen-2.5-1B-RLCD, has been released to achieve high-efficiency inference in Apple Silicon environments. This repository adopts Qwen/Qwen2.5-1.5B-Instruct as its base model and provides an inference engine incorporating parallel constrained decoding.

Traditional autoregressive decoding generates tokens one by one in sequence, which causes latency to increase as schemas grow larger. By evaluating multi-field JSON schemas and choices (enum, boolean) in parallel, this method achieves a 5.6x to 7.0x latency reduction and 100% schema validity on an Apple Silicon M4 Max.

Specifications

• Base Model: Qwen/Qwen2.5-1.5B-Instruct
• Parameter Count: 1.54B (Non-embedding parameters: 1.31B)
• Context Length: Full 32,768 tokens (up to 8,192 generation tokens)
• License: apache-2.0

Performance

According to measurements published by the author using mlx-community/Qwen2.5-1.5B-Instruct-4bit on an Apple Silicon M4 Max (macOS Sequoia) environment, the comparison between the standard autoregressive baseline and parallel constrained decoding is as follows:

The author’s measurements show that the effect of parallel processing becomes more pronounced in scenarios with an increasing number of target fields. For “Enterprise Support Triage," which extracts 28 fields, conventional autoregressive generation required 1,900 ms, whereas this method achieved a significant speedup of 7.0x at 270 ms.

Additionally, consistent speedups of 5.6x (75 ms to 89 ms) were recorded for the high-cardinality “High-Cardinality Tariff" with 255 choices, as well as the 4-field “Fintech Fraud Routing" and “Code Security Audit." In all test cases, syntax validity is 100% guaranteed, and the strength lies in executing high-speed processing while completely preventing grammar errors and missing fields during structured data output.

Strengths and Use Cases

Building upon the high instruction-following and structured data comprehension capabilities of the base model Qwen/Qwen2.5-1.5B-Instruct, this combines parallel constrained decoding technology to exhibit specialized strengths in information extraction and classification tasks.

Specifically, input context and schema definition text are preloaded (prefilled) into the MLX KV Cache (Key-Value Cache) just once, and that state is broadcast in parallel to the evaluation of each field. By applying mask extraction (logit slicing) only to token IDs permitted for each field to directly calculate probabilities from predefined choices (booleans or up to 255 enums), unnecessary token generation loops are avoided.

This makes it highly effective in business scenarios such as:
– Fintech fraud routing (identifying risk levels and determining the necessity of confirmation)
– Security audits (classifying vulnerable code and assigning remediation priorities)
– Automated triage of large-scale customer support tickets (department routing and urgency setting)
– Identifying product codes or tariff classifications (such as HS codes) with numerous classification candidates

Since JSON syntax breaks or unnecessary key outputs (hallucinations) do not occur by design, it is extremely well-suited for integration with backend systems and the construction of automated pipelines where accuracy is required.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 1.5B parameters (taken from the base model Qwen/Qwen2.5-1.5B-Instruct)

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Recent Models in the Same Size Class

Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.

How to Get It

This model and inference engine are available through the MLX library. Setup procedures and dependency installations can be executed with the following commands:

git clone https://github.com/your-org/parallel-constrained-decoding.git
cd parallel-constrained-decoding

python3 -m venv.venv
source.venv/bin/activate
pip install -r requirements.txt

From Python code, you can use StructuredSchema to define schemas and call the run_parallel_generation function to execute parallel high-speed inference. You can also launch the web interface (python3 -m uvicorn server.app:app --host 0.0.0.0 --port 8000) to check latency comparisons with conventional autoregressive generation and structured data output right in your browser.

Sources

• https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD