{"id":2486,"date":"2026-09-22T20:13:54","date_gmt":"2026-09-22T11:13:54","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/22\/transformers-llama-cpp-gguf-support\/"},"modified":"2026-09-22T20:13:54","modified_gmt":"2026-09-22T11:13:54","slug":"transformers-llama-cpp-gguf-support","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/22\/transformers-llama-cpp-gguf-support\/","title":{"rendered":"Run GGUF Models Directly in Transformers with llama.cpp Support"},"content":{"rendered":"

<\/p>\n

At a Glance<\/h2>\n
\n\n\n\n\n\n\n\n
Item<\/th>\nValue<\/th>\n<\/tr>\n<\/thead>\n
Publisher<\/td>\nHugging Face Blog<\/td>\n<\/tr>\n
Published<\/td>\n2026-09-22<\/td>\n<\/tr>\n
Source type<\/td>\nPrimary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Values determined by this site’s code at collection time. Dates are JST.<\/em><\/p>\n

<\/p>\n

Overview<\/h2>\n

On September 22, 2026, Hugging Face announced the addition of direct and efficient execution support for GGUF format models from llama.cpp<\/code> within the transformers<\/code> inference library. This enables users to load GGUF checkpoints from the Hub via the from_pretrained<\/code> API and run inference directly on their local machines. Initial optimization targets local inference of the Qwen3.5 architecture on Apple Silicon environments.<\/p>\n

Announcement Details<\/h2>\n

Hugging Face announced a new integration allowing GGUF format models to be loaded and executed directly within the transformers<\/code> ecosystem. GGUF is a format developed by the llama.cpp<\/code> team and widely adopted in local AI tools such as Ollama, LM Studio, and Jan. This addition allows developers to use lightweight GGUF models while maintaining their familiar transformers<\/code> Python APIs and PyTorch workflows.<\/p>\n

In terms of performance, aiming for execution speeds close to llama.cpp<\/code>, a mechanism was introduced to directly utilize ggml<\/code> Metal kernels via the kernels<\/code> library. Specifically, it leverages ggml-quantization<\/code> for matrix operations, ggml-norm<\/code> for normalization, ggml-attn<\/code> for attention processing, and ggml-gated-delta-net<\/code> for hybrid architectures, alongside a custom topk<\/code> kernel that speeds up routing in MoE models. Processing weights on Metal while keeping them packed reduces memory consumption and overhead, though it automatically falls back to sdpa<\/code> (Scaled Dot-Product Attention) if compatible kernels are not available.<\/p>\n

For usage forms, beyond loading from scripts, it also supports launching an OpenAI-compatible API server using the transformers serve<\/code> command. This enables client applications like Jan and Pi to use models via local endpoints such as http:\/\/localhost:8000\/v1<\/a>. Additionally, for those wishing to perform fine-tuning, setting GgufConfig(dequantize=True)<\/code> dequantizes the model to incorporate it into standard transformers<\/code> training workflows.<\/p>\n

Furthermore, to reduce synchronization between the CPU and GPU\u2014which often becomes an inference bottleneck\u2014improvements were also made to the generate<\/code> loop itself. Enhancements include early filtering of unnecessary attention masks in inputs without padding (PR #48814) and asynchronous stop condition determination during token generation to increase execution overlap (PR #47975). These optimizations also contribute to improving inference efficiency for all non-GGUF transformers<\/code> models.<\/p>\n

Note that the initial implementation currently has several limitations. The fast inference path using packed kernels is limited to Apple Silicon (MPS) environments, and other devices may require weight expansion. Additionally, optimizations for batched processing with padding (generate_batch<\/code>) and support for model architectures other than Qwen3.5 (Dense\/MoE) and Qwen3.8 are planned for gradual expansion as future tasks.<\/p>\n

Background<\/h2>\n

When the GGML and llama.cpp<\/code> projects joined Hugging Face, their complementary roles were highlighted: llama.cpp<\/code> provides the foundation for efficient local inference, while transformers<\/code> provides the foundation for model definitions. llama.cpp<\/code> features a dedicated runtime, memory management, and extensive hardware support, positioning it as the recommended engine for scenarios where efficient local inference is the top priority.<\/p>\n

With GGUF format models now directly executable within transformers<\/code>, the distance between these two foundations has narrowed. By closely combining the flexible model definitions and ecosystem provided by transformers<\/code> with ggml<\/code>‘s high-performance optimization kernels, developers can simultaneously advance local inference and advanced customization\/verification within a single environment.<\/p>\n

Impact on Local LLM Users<\/h2>\n

For developers and engineers operating open-weight models in local environments such as personal PCs or servers, this integration brings the significant advantage of leveraging GGUF checkpoints while maintaining standard Python and PyTorch workflows. The specific impacts and utilization points presented in the documentation are as follows:<\/p>\n

    \n
  1. \n

    Flexible Experimentation and Verification in Python\/PyTorch Environments<\/strong>
    \nUsing familiar PyTorch toolkits, developers can inspect intermediate activations via hooks, directly modify the model’s forward pass, and prototype custom layers for GGUF models. It also makes it easy to experiment with incorporating custom logits processors or stop conditions into generate<\/code>, or writing custom generation loops in Python.<\/p>\n<\/li>\n

  2. \n

    Evaluation and Fine-Tuning in Existing Workflows<\/strong>
    \nExisting transformers<\/code> evaluation workflows can be applied as-is to measure generation quality across checkpoints of various quantization types. Furthermore, by loading the original checkpoint and the GGUF-converted model simultaneously on transformers<\/code>, conversion accuracy can be verified while accounting for quantization errors. For training, users can specify GgufConfig(dequantize=True)<\/code> to dequantize weights and proceed to standard transformers<\/code> fine-tuning tasks.<\/p>\n<\/li>\n

  3. \n

    Utilization of Kernels Beyond the GGUF Format and Expansion to Other Modalities<\/strong>
    \nThis optimization goes beyond merely reading GGUF files. Because ggml<\/code>‘s optimization kernels (matrix operations, normalization, attention, etc.) function on a per-tensor basis, they can potentially accelerate processing via PyTorch even when the entire model is not provided in GGUF format, or for new architectures, research models, and custom variants lacking full implementations on the llama.cpp<\/code> side. As a future prospect, applying these kernels to other modalities such as image, audio, and multimodal models is also envisioned.<\/p>\n<\/li>\n

  4. \n

    Current Limitations and Feedback Acceptance<\/strong>
    \nAt present, the speedup path using packed kernels is limited to Apple Silicon (MPS) environments; when compatible kernels are unavailable, it falls back to weight expansion (dequantization), increasing memory consumption. Moreover, optimizations for batched processing with padding (generate_batch<\/code>) and support for architectures other than Qwen3.5 (Dense and MoE) or Qwen3.8 are noted as future tasks. Hugging Face invites users to share checkpoints and use cases on GitHub Issues if there are GGUF models they wish to use in local environments, planning to determine priority and progressively expand support based on demand.<\/p>\n<\/li>\n<\/ol>\n

    <\/p>\n

    Related Articles<\/h2>\n