Hugging Face AI Updates: September 16, 2026

1. IBM Research Published a Consistency Analyzer That Halved the Gap Between Mean@k and Pass^k

Hugging Face. The post targets a metric most agent evals hide: Mean@k reports the average pass rate across runs, while Pass^k requires success on all k runs, and the spread between them is what production actually feels. ALTK-Evolve’s Consistency Analyzer finds flip-prone decision points by resampling k completions (default k=5) at each step, costing one extra model call per decision step offline rather than replaying full task executions, then generates targeted guidelines from the unstable steps. On ReAct agents running GPT-4.1 against AppWorld, the consistency gap fell from 24.4 to 12.0 percentage points and Pass^5 rose from 53.0% to 69.0% while Mean@5 held at 77.4% to 81.0%, with a +13.0pp transfer gain when the guidelines were applied to related tasks. The toolkit is open source at github.com/AgentToolkit/altk-evolve. Source