Transformers Adds Fast GGUF Model Inference on Apple Silicon
Transformers adds efficient GGUF inference on Apple Silicon, using ggml kernels to run quantized Qwen3.5 and Qwen3.8 models through familiar APIs.
Summary
Hugging Face added GGUF support to Transformers on September 22, 2026, letting Apple Silicon users load quantized Hub checkpoints through from_pretrained and standard generation APIs. The initial packed MPS path supports Qwen3.5 dense and mixture-of-experts models plus compatible Qwen3.8 checkpoints. It reuses ggml Metal kernels for quantization, normalization, attention and gated delta networks, alongside a custom top-k kernel, while generate changes reduce CPU and GPU synchronization for all Transformers models.
GGUF combines weights, tokenizer metadata and optional chat templates in one file. Unsloth’s Qwen3.5-4B shrinks from 8.42 GB in BF16 to 3.53 GB with Q6_K, 3.14 GB with Q5_K_M or 2.74 GB with the recommended Q4_K_M. On a 32 GB M2 Max MacBook Pro running macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0, Transformers approached llama.cpp throughput across three checkpoints, although its test included prefill while llama-bench measured decode only. Transformers can also serve GGUF through an OpenAI-compatible API for clients including Jan and Pi. llama.cpp remains recommended for maximum local efficiency and wider hardware support. Packed inference is currently MPS-only, batching and padding need optimization, and architecture coverage will expand gradually.
Positives
- Q4_K_M reduces Unsloth’s Qwen3.5-4B from 8.42 GB in BF16 to 2.74 GB, making local inference practical on smaller-memory Macs.
- Standard from_pretrained and generate APIs now load and run compatible GGUF checkpoints without a separate inference runtime.
- ggml quantization, normalization, attention and gated delta network kernels keep packed weights on Metal and reduce per-token GPU work.
- Transformers serve exposes GGUF models through an OpenAI-compatible endpoint usable by Jan, Pi and other compatible clients.
- Generate optimizations reduce synchronization overhead for all supported Transformers models, not only GGUF workloads.
- Qwen3.5 dense and mixture-of-experts models, plus compatible Qwen3.8 checkpoints, receive initial packed-kernel support.
Risks & concerns
- Packed GGUF inference is currently limited to Apple Silicon through MPS, despite GGUF import by dequantization remaining available elsewhere.
- Padded batches cannot use the new attention-mask shortcut and may perform worse until generate_batch receives further MPS optimization.
- Architecture support currently covers only Qwen3.5 dense and mixture-of-experts models plus compatible Qwen3.8 checkpoints.
- Missing compatible quantization kernels force Transformers to dequantize weights, increasing memory use.
- Benchmark conditions differ because Transformers includes prefill while llama-bench reports decode-only throughput.
- llama.cpp still offers broader hardware support, dedicated memory management and greater efficiency for local inference.