Ai2's StateBench shows frontier models losing track after thousands of state changes
Ai2 researchers Jackson Petty, William Merrill and Ashish Sabharwal tested four levels of state tracking; frontier models scored 13% or lower on the longest general finite tasks.
By RuntimeWire Staff · Published
Primary source: Hugging Face Newsroom
Why it matters
Long context claims usually rest on retrieval, which StateBench treats as only the easiest level of tracking. The benchmark shows frontier models failing dependent updates at long lengths, while directly trained Gated DeltaNet and hybrid models generalized far beyond their training sequences. That distinction affects architecture choices, pretraining data and any product that depends on a model following code, logs or agent state over time.
Jackson Petty, William Merrill (@lambdaviking) and Ashish Sabharwal released StateBench on November 3rd. The benchmark tests whether a language model can follow information as it changes across a long document and where it starts losing the thread.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.