The development

What happened

Ai2 introduced StateBench on 3 November to test whether language models track changing values over long inputs. Its four levels progress from fixed assignments and latest-value retrieval to cumulative updates and dependencies between variables.

Key facts

  • The authors report accuracy of 13% or less for tested frontier models on the longest general-finite tasks. This is not overall model accuracy.

  • Separately trained 275-million-parameter architectures saw up to 256 assignments in training and up to 64,000 in testing.

  • Recurrent and hybrid models generalized strongly on the three easier levels. The most flexible Gated DeltaNet variant stayed near-perfect on the hardest level to roughly ten times its training length.

  • Transformers degraded on longer sequences; pretrained models were sensitive to presentation format. The announcement links data, code and a technical report.

  • The synthetic tasks keep the number of variables fixed, excluding new entities appearing during a conversation.

The context

Why it matters

TideReach assessment: recall tests alone may miss failures to follow chained updates. These results help distinguish task types, but synthetic, fixed-variable tests cannot establish reliability in real conversations; the reported experiments have not been independently reproduced here.