StateBench Finds GPT-6, Claude and Gemini Falter on Long State Tracking
StateBench finds recurrent and hybrid models outlast transformers on long state tracking, while frontier AI falls to 13% on the hardest tests as inputs grow.
Summary
Ai2 released StateBench, its data, code and technical report on November 3, 2026, to measure how language models track changing entities across long documents. Its Krohn, Rhodes based hierarchy spans four levels: R-trivial immutable assignments, aperiodic latest-value updates, soluble self-referential updates, and general finite cross-variable references, including the five-object S₅ swapping problem. Matched tests preserve variables, values and update positions while changing assignment types and formats.
Open pretrained models included the 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, plus 275-million and 2.7-billion-parameter models trained on identical data with contexts no longer than 64,000 tokens. Most handled R-trivial tasks, weakened on longer aperiodic inputs and remained near chance on general finite tasks. Format mattered: 275-million-parameter hybrid and Gated DeltaNet models approached the ceiling in RULER-style tests but neared chance on equivalent spreadsheet edits.
Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna performed well on R-trivial and aperiodic tasks, but fell to 13% accuracy or less on the longest general finite tests. Six 275-million-parameter models trained on up to 256 assignments and tested on 64,000, or 250 times longer, showed hybrids and Gated DeltaNet generalizing almost perfectly through soluble tasks; transformers degraded and never learned general finite tasks. A flexible Gated DeltaNet stayed near perfect to about 10 times training length on the hardest level. Open questions include limited memory updates, numerical precision, attention integration and tracking newly introduced variables.
Positives
- StateBench releases its benchmark, data, code and technical report for reproducible evaluation across four theoretically grounded difficulty levels.
- Gated DeltaNet and hybrid models trained directly on StateBench generalized almost perfectly across tens of thousands of R-trivial, aperiodic and soluble assignments.
- A flexible Gated DeltaNet remained near perfect on general finite sequences about 10 times longer than its maximum training examples.
- Six 275-million-parameter models were tested at 64,000 assignments, 250 times their training limit, exposing architectural differences that shorter evaluations can hide.
Risks & concerns
- Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna dropped to 13% accuracy or less on the longest general finite tests.
- Transformers degraded as directly trained sequences lengthened and never learned StateBench’s hardest general finite level.
- 275-million-parameter hybrid and Gated DeltaNet models shifted from near-ceiling to near-chance performance when equivalent tasks changed from RULER-style prompts to spreadsheets.
- StateBench assumes a fixed variable count, leaving real tasks with newly appearing people, objects or other entities outside its current framework.
- Gated DeltaNet’s capabilities under limited memory updates, different numerical precision and attention integration remain unresolved.