Hugging Face AI Updates: September 22, 2026

1. tokenizers v1 Encodes 3 to 30 Times Faster With an Unchanged API

Hugging Face published the tokenizers v1 release candidate, measuring 3x to 30x faster encoding than v0.23 single-threaded on an Apple M4 Max, and 76 percent of linear scaling across eight workers. The API, vocabulary, merge ranks, and resulting token IDs are all unchanged, so v1 is a drop-in swap. The speedup comes from bitstream-based splitting that replaces the regex engine, thread-local word caching, reusable scratch buffers instead of per-call allocation, batched model processing, and direct byte writing on decode. The RC is on crates.io via cargo add tokenizers --pre, with 1.0.0 pending broader model family support. Source

2. A Physics Framing Turns Block Pruning Into an Ising Optimization Problem

A post on the Hugging Face blog from Multiverse Computing reformulates LLM block removal as an Ising optimization problem, treating which transformer blocks to prune as a spin configuration to minimize rather than a greedy layer-by-layer decision. Casting pruning as a global combinatorial objective is what separates this from the usual sequential heuristics, since block importance is not independent across layers. Source