Hugging Face added support on September 22, 2026, for running quantized GGUF models directly inside its transformers library. The integration targets local inference by reusing underlying ggml Metal kernels through the Python kernels library. Publishers including Unsloth, LM Studio Community, and bartowski distribute these checkpoints on the Hub, where GGUF files have recorded millions of downloads. The new loader allows models to run on Apple Silicon hardware without converting or expanding weights prior to execution.
Initial support focuses on the Qwen3.5 architecture, spanning both dense and mixture-of-experts variants alongside compatible Qwen3.8 checkpoints. Loading a model requires passing the Hub repository identifier and the specific GGUF filename into the standard from_pretrained method. When weights remain packed on Metal, transformers automatically loads compatible ggml layer kernels and applies ggml-org/ggml-attn for attention operations. If that attention kernel is unavailable, the software issues a warning and falls back to PyTorch's native scaled dot-product attention.
The GGUF format packages model weights, tokenizer metadata, and chat templates within a single file. Quantization reduces memory requirements by lowering numerical precision across specific tensors. In Unsloth's Qwen3.5-4B release, a full 16-bit brain floating-point checkpoint measures 8.42 gigabytes. Quantizing to Q6_K reduces the file to 3.53 gigabytes, Q5_K_M brings it to 3.14 gigabytes, and Q4_K_M reduces the file size to 2.74 gigabytes. For model fine-tuning, developers can dequantize weights into 16-bit brain floating point using GgufConfig.
Specialized Metal kernels
Five specialized Metal kernels handle model execution on Apple hardware. A quantization kernel reads packed weights directly for matrix multiplications, including expert layers within mixture-of-experts architectures. A fused normalization kernel handles zero-centered RMSNorm operations for Qwen3.5 and Qwen3.8 models. Attention relies on ggml's Metal flash attention kernel, while a dedicated kernel accelerates the gated delta network used in hybrid linear-attention layers. Hugging Face also built a custom Metal top-k routing kernel that combines softmax and expert selection to resolve bottlenecks in mixture-of-experts models.
The implementation also integrates with transformers serve to expose an OpenAI-compatible HTTP endpoint at localhost port 8000. Users specify the model repository and filename using a colon separator, allowing clients like Jan or Pi to connect directly. The serving command accepts reasoning flags to turn thinking outputs on, off, or keep the default automatic behavior defined by the model's chat template. Without compatible quantization kernels, the system falls back to dequantizing the entire model, consuming additional memory.
Generation loop changes
Two modifications to the generate function reduce delays between the CPU and GPU. Pull request 48814 removes unnecessary padding masks early during generation for unpadded decoder-only inputs, preserving causal attention without repeatedly inspecting an all-ones mask. Pull request 47975 defers stopping checks by copying stopping decisions asynchronously and evaluating them on the subsequent decoding step. These modifications allow the CPU to continue scheduling tasks while the GPU executes, improving generation performance across all transformers models.
Benchmarks compared transformers against the llama-bench tool using an Apple MacBook Pro equipped with an M2 Max processor, 32 gigabytes of unified memory, macOS 26.6, PyTorch 2.12.1, and kernels 0.17.0. The test measured token generation across three repetitions of 128 decoded tokens after a 12-token prompt, cooling the machine for 90 seconds between runs. The llama-bench evaluation used release b10200 and ggml 0.18.0 to measure decode throughput without prompt processing. Hugging Face reported that transformers achieved performance close to llama.cpp across dense and mixture-of-experts checkpoints, despite the transformers measurement including prompt prefill time.
The organization noted that ggml kernels can extend beyond the GGUF file format to architectures that llama.cpp does not implement. Because transformers already contains PyTorch implementations of various models, developers can accelerate operations without rewriting complete architectures in C++. Hugging Face stated this approach creates a path to accelerate research models and other modalities, such as computer vision, audio, and multimodal networks, directly within PyTorch.
Scope and limitations
The packed inference loader remains subject to operational constraints. The accelerated execution path functions exclusively on Apple Silicon via Metal Performance Shaders, with padded batching currently lacking the optimizations applied to single unpadded inputs. Architecture support remains limited to Qwen3.5 and Qwen3.8 checkpoints, though Hugging Face stated it plans to expand coverage based on user requests submitted to its issue tracker. Arthur Zucker initiated the project and reviewed pull requests, Cyril Vallez authored the generate changes, and Lysandre Debut oversaw the engineering effort.
