The development
What happened
Ai2 introduced StateBench on 3 November to test whether language models track changing values over long inputs. Its four levels progress from fixed assignments and latest-value retrieval to cumulative updates and dependencies between variables.
Key facts
The authors report accuracy of 13% or less for tested frontier models on the longest general-finite tasks. This is not overall model accuracy.
Separately trained 275-million-parameter architectures saw up to 256 assignments in training and up to 64,000 in testing.
Recurrent and hybrid models generalized strongly on the three easier levels. The most flexible Gated DeltaNet variant stayed near-perfect on the hardest level to roughly ten times its training length.
Transformers degraded on longer sequences; pretrained models were sensitive to presentation format. The announcement links data, code and a technical report.
The synthetic tasks keep the number of variables fixed, excluding new entities appearing during a conversation.
The context
Why it matters
TideReach assessment: recall tests alone may miss failures to follow chained updates. These results help distinguish task types, but synthetic, fixed-variable tests cannot establish reliability in real conversations; the reported experiments have not been independently reproduced here.