Hugging Face AI Updates: November 4, 2026

1. Hugging Face Proposes Model Integration Contracts, CPU-Sized Fixtures That Would Have Caught a Two-Month UMT5 Regression

Hugging Face introduced transformers-mic, a proposed standard for fast integration tests in Transformers, which supports more than 750 model types and runs 88,000+ CPU tests on main plus 200,000+ GPU tests in a daily run. A Model Integration Contract (MIC) pairs a fixture (a shrunk model built from the real checkpoint’s config, with fewer and narrower layers but every layer kind, byte-identical tokenizer and processor files, and the original weight file format), a runnable Markdown usage path such as generation or a training step, and human-reviewed baselines; contracts are pinned to frozen revisions in a fleet dataset on the Hub and picked per PR from the files a patch changes. Replaying frozen MICs over five months of commits, a 66 MB UMT5 fixture (from a 1,227 MB checkpoint) flagged, on every commit, the July 30 SDPA change that let UMT5’s decoder attend to future tokens until its September 28 fix, while an eager-attention-only variant missed it; with the fix reverted on a proof branch, the T5 contract’s check ran in 13.4 seconds on CPU. The shrink recipe keeps mixed layer types such as sliding and full attention or dense and MoE layers (Llama 4 Scout drops from 108.6 billion to 44.7 million parameters), and the tmic CLI has an LLM draft new contracts through Inference Providers, billed to the user’s account, while changed baselines require human review. Overfitting a Whisper fixture on a few examples raised its catch rate on injected faults from 18 of 24 to 24 of 24, though both versions missed an assisted-decoding bug that only the real checkpoint exposed. As of November 1 the MIC job reports failures in Transformers CI without blocking PRs, and coverage across model families is still partial. Source

2. Ai2’s StateBench Finds Frontier Models Fall to 13% or Lower on Long, Hard State-Tracking Problems

Ai2 released StateBench (dataset allenai/state-bench, plus a tech report and code), a benchmark for how well language models track entities’ changing values through long documents. Grounded in Krohn-Rhodes theory, it has four levels framed as variable assignment: r-trivial (immutable values), aperiodic (reassignment, where only the latest value matters), soluble (self-referential updates such as x = x + 5) and general finite (assignment by reference, such as swaps, including the S5 permutation problem), all derived from the same assignment sequence so that only the update type changes. Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the first two levels well, but on soluble and general finite problems accuracy fell sharply with length, to 13% or lower on the longest general finite problems. Open pretrained models, including the 7B Olmo 3 transformer and the 7B Olmo Hybrid with Gated DeltaNet layers (none trained on contexts beyond 64K tokens), stayed at chance on general finite problems. When 275M-parameter models were trained from scratch on sequences of up to 256 assignments and tested on up to 64,000, Gated DeltaNet and hybrid models generalized almost perfectly on the three easier levels, transformers degraded with length and never learned general finite problems, and a Gated DeltaNet variant with a more flexible memory update stayed near perfect on the hardest level out to about 10 times its training length. Source

3. CWE-bench v2 Opens 100 Find-and-Fix Security Tasks Across 91 CWEs, With Gemini 4 Argon and Claude Opus 5.5 Nearly Tied

Collinear AI, with Hugging Face’s Nathan Habib among the authors, published CWE-bench v2 on the Hugging Face blog: a public set of 100 tasks covering 91 CWEs in which an agent gets a checkout of a real open-source repository and a short instruction and must find and fix a security weakness without breaking existing functionality. Scoring is programmatic and deterministic, and a held-out set is kept for tracking progress on unseen tasks (evaluated on request). Every task was reviewed by hand and checked against criteria including that its verifier tests behavior rather than the structure of one fix and rejects memorized solutions such as the known upstream patch. Of 11 models evaluated, Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% and 78.0%; 95 tasks are solved by at least one model, yet the leader fails 19% of the set on every attempt. On cost, DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout, Opus 5.5 adds about ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92. Each task is a self-contained Harbor task, so the gated collinear-ai/cwe-bench dataset runs straight from the Hub with harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4; open-weight results are added by PR on the model’s repo, API-only models through a leaderboard Space, and the tasks are tagged as RL environments for training defensive agents. Source

4. IBM Research’s Palette Fine-Tunes a 20B Open-Weight Model to Build Slide Decks in a House Design Language

IBM Research described Palette, a single LoRA adapter on a 20B open-weight model (the post names neither the base model nor a release of the adapter) that plays two roles: a designer that turns a plan into a structured JSON brief with palette, type system and per-slide layout, content and rationale, and a coder that writes pptxgenjs JavaScript for one slide at a time, so slide calls run in parallel and the coding stage is about 6 to 7 times faster. Training data was about 1,000 synthetic decks (roughly 10,000 slides) generated by a large open teacher following the IBM Design Language, with counterfactual reasoning traces and every deck rendered and checked for overlap and overflow; IBM found that fixing the reference decks in the teacher’s prompt changed outputs far more reliably than adding rules, and that coder examples, about five times longer, forced a 5x oversampling of designer examples. Users steer slides with inline [[ ]] directives from 12 categories (for example [[make this a timeline]] or [[no pricing]]), and a check-and-repair harness applies fix rules, renders each slide, flags overlap and clipping, and keeps an editor model’s rewrite only when the checker confirms an improvement, adding 3.5 points on average across 131 PresentBench decks. On PresentBench, Palette ranks first on Economics and Education, third on Talk, and second on the three-domain mean (59.7 against NotebookLM’s 60.8); IBM plans to use the harness checks as rewards for GRPO-style training next. Source