Run GGUF Models Directly in Transformers with llama.cpp Support

September 22, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

On September 22, 2026, Hugging Face announced the addition of direct and efficient execution support for GGUF format models from llama.cpp within the transformers inference library. This enables users to load GGUF checkpoints from the Hub via the from_pretrained API and run inference directly on their local machines. Initial optimization targets local inference of the Qwen3.5 architecture on Apple Silicon environments.

Announcement Details

Hugging Face announced a new integration allowing GGUF format models to be loaded and executed directly within the transformers ecosystem. GGUF is a format developed by the llama.cpp team and widely adopted in local AI tools such as Ollama, LM Studio, and Jan. This addition allows developers to use lightweight GGUF models while maintaining their familiar transformers Python APIs and PyTorch workflows.

In terms of performance, aiming for execution speeds close to llama.cpp, a mechanism was introduced to directly utilize ggml Metal kernels via the kernels library. Specifically, it leverages ggml-quantization for matrix operations, ggml-norm for normalization, ggml-attn for attention processing, and ggml-gated-delta-net for hybrid architectures, alongside a custom topk kernel that speeds up routing in MoE models. Processing weights on Metal while keeping them packed reduces memory consumption and overhead, though it automatically falls back to sdpa (Scaled Dot-Product Attention) if compatible kernels are not available.

For usage forms, beyond loading from scripts, it also supports launching an OpenAI-compatible API server using the transformers serve command. This enables client applications like Jan and Pi to use models via local endpoints such as http://localhost:8000/v1. Additionally, for those wishing to perform fine-tuning, setting GgufConfig(dequantize=True) dequantizes the model to incorporate it into standard transformers training workflows.

Furthermore, to reduce synchronization between the CPU and GPU—which often becomes an inference bottleneck—improvements were also made to the generate loop itself. Enhancements include early filtering of unnecessary attention masks in inputs without padding (PR #48814) and asynchronous stop condition determination during token generation to increase execution overlap (PR #47975). These optimizations also contribute to improving inference efficiency for all non-GGUF transformers models.

Note that the initial implementation currently has several limitations. The fast inference path using packed kernels is limited to Apple Silicon (MPS) environments, and other devices may require weight expansion. Additionally, optimizations for batched processing with padding (generate_batch) and support for model architectures other than Qwen3.5 (Dense/MoE) and Qwen3.8 are planned for gradual expansion as future tasks.

Background

When the GGML and llama.cpp projects joined Hugging Face, their complementary roles were highlighted: llama.cpp provides the foundation for efficient local inference, while transformers provides the foundation for model definitions. llama.cpp features a dedicated runtime, memory management, and extensive hardware support, positioning it as the recommended engine for scenarios where efficient local inference is the top priority.

With GGUF format models now directly executable within transformers, the distance between these two foundations has narrowed. By closely combining the flexible model definitions and ecosystem provided by transformers with ggml's high-performance optimization kernels, developers can simultaneously advance local inference and advanced customization/verification within a single environment.

Impact on Local LLM Users

For developers and engineers operating open-weight models in local environments such as personal PCs or servers, this integration brings the significant advantage of leveraging GGUF checkpoints while maintaining standard Python and PyTorch workflows. The specific impacts and utilization points presented in the documentation are as follows:

1. Flexible Experimentation and Verification in Python/PyTorch Environments
   Using familiar PyTorch toolkits, developers can inspect intermediate activations via hooks, directly modify the model’s forward pass, and prototype custom layers for GGUF models. It also makes it easy to experiment with incorporating custom logits processors or stop conditions into generate, or writing custom generation loops in Python.

2. Evaluation and Fine-Tuning in Existing Workflows
   Existing transformers evaluation workflows can be applied as-is to measure generation quality across checkpoints of various quantization types. Furthermore, by loading the original checkpoint and the GGUF-converted model simultaneously on transformers, conversion accuracy can be verified while accounting for quantization errors. For training, users can specify GgufConfig(dequantize=True) to dequantize weights and proceed to standard transformers fine-tuning tasks.

3. Utilization of Kernels Beyond the GGUF Format and Expansion to Other Modalities
   This optimization goes beyond merely reading GGUF files. Because ggml's optimization kernels (matrix operations, normalization, attention, etc.) function on a per-tensor basis, they can potentially accelerate processing via PyTorch even when the entire model is not provided in GGUF format, or for new architectures, research models, and custom variants lacking full implementations on the llama.cpp side. As a future prospect, applying these kernels to other modalities such as image, audio, and multimodal models is also envisioned.

4. Current Limitations and Feedback Acceptance
   At present, the speedup path using packed kernels is limited to Apple Silicon (MPS) environments; when compatible kernels are unavailable, it falls back to weight expansion (dequantization), increasing memory consumption. Moreover, optimizations for batched processing with padding (generate_batch) and support for architectures other than Qwen3.5 (Dense and MoE) or Qwen3.8 are noted as future tasks. Hugging Face invites users to share checkpoints and use cases on GitHub Issues if there are GGUF models they wish to use in local environments, planning to determine priority and progressively expand support based on demand.

Related Articles

• Pruning LLMs Like a Physicist: Constrained Binary Optimization

Sources

• https://huggingface.co/blog/transformers-llama-cpp-quants