Hugging Face released a release candidate for version 1 of its tokenizers library on crates.io on September 21, 2026. The release candidate produces the same token IDs, API interfaces, vocabulary, and merge ranks as version 0.23 while overhauling the underlying encoding engine.
Tests run on an Apple M4 Max chip showed single-threaded encoding executing 3 to 30 times faster than version 0.23 across ten model families. Hugging Face recorded the low end on t5-base and the high end on gpt2. The library scaled at 76 percent of linear performance across eight physical CPU cores running distinct documents.
SIMD splitting and memory
Developers replaced regex splitting on byte-pair encoding paths with bitcannon, a routine using SIMD bitstream operations. The system processes 64 bytes per register operation across patterns used by GPT-2, cl100k, o200k, Tekken, and DeepSeek. Tokenizers using unrecognized patterns retain the regular expression path.
The project divided the original crate into a modular workspace, isolating the tk-encode runtime from optional serialization, conversion, and training components. A thread-local cache named WordCache saves pre-token results to avoid repeat merges on identical words. For the merge loop, caller-owned scratch buffers eliminate repeated heap allocations, and candidate pairs are packed into 64-bit values to remove comparison branching.
Roadmap to 1.0.0
Contributors from IBM, NVIDIA, and the ExecuTorch team assisted Hugging Face by submitting patches and testing across varied hardware. Before issuing 1.0.0, the team intends to apply tk-encode to training validation, add bitnorm support, and simplify the Python bindings. Later plans include developing tok-devices to evaluate GPU-side encoding and batch decoding.
