IBM ALTK-Evolve Study: Agent Memory Must Match Model Capability
IBM's ALTK-Evolve study finds agent memory must match model capability, lifting gpt-oss-120b completion 16.1 points with just 5% more tokens on AppWorld.
Summary
Following an earlier ACE comparison, IBM Research’s Vatche Isahagian, Gaodan Fang, Jayaram Radhakrishnan, Punleuk Oum, Ashwath Vaithinathan Aravindan, Evelyn Duesterwald, G Thomas, Vinod Muthusamy, Merve Unuvar and Ayhan Sebin published results on Hugging Face on August 18, 2026. ALTK-Evolve converts successful and failed agent trajectories into reusable guidelines without weight updates, human annotation or test leakage. Eight models, from a 30B dense model to proprietary frontier systems, ran 585 AppWorld tasks, 168 test_normal and 417 test_challenge, across nine simulated apps including calendars, messaging and payments. Baseline used no memory; full memory injected every training-mined guideline each ReAct step; curated retrieval combined a fixed high-confidence core with task-specific guidelines.
On test_normal, gpt-oss-120b, a 117B MoE, rose from 39.9% TGC and 21.4% SGC to 56.0% and 37.5% with retrieval, gaining 16.1 points on both. Full sets moved DeepSeek-V3.2, a 671B MoE, from 79.8/64.3 to 89.3/80.4, up 9.5/16.1; Claude Opus 4.6 from 90.5/87.5 to 94.6/94.6, up 4.1/7.1; and GPT-5.5 from 92.3/82.1 to 95.2/89.3, up 2.9/7.2. GLM-5, a 745B MoE, stayed at 87.5/80.4. TGC measures completed tasks; stricter SGC requires every scenario variant to pass.
Strong models with headroom favored full sets, weaker models selective retrieval, and saturated GLM-5 no extra context. DeepSeek tokens rose from 148K to 263K, 78%, while ReAct steps stayed near 18 to 19; gpt-oss rose from 110K to 166K, 51%, with full memory but only to 116K, 5%, with retrieval. Stable-prefix prompt caching could cut full-set cost. IBM plans an outcome-trained selector beyond cosine similarity, teacher-distilled memory for very weak models, broader testing beyond AppWorld and controlled context-window experiments; the ALTK-Evolve library and technical report are available.
Positives
- gpt-oss-120b gained 16.1 percentage points in TGC and SGC with curated retrieval while using only 5% more tokens.
- DeepSeek-V3.2 gained 9.5 TGC points and 16.1 SGC points from its full self-mined guideline set.
- Claude Opus 4.6 and GPT-5.5 gained 7.1 and 7.2 SGC points despite already high baseline scores.
- ALTK-Evolve learns from successful and failed trajectories without model weight updates, human annotation or AppWorld test data.
- Prompt caching can reuse the static guideline prefix across ReAct steps, reducing the effective production cost of full memory.
Risks & concerns
- GLM-5 recorded no measurable improvement, remaining at 87.5% TGC and 80.4% SGC with full memory.
- DeepSeek-V3.2’s full guideline set increased average token use from 148K to 263K per task, a 78% overhead.
- gpt-oss-120b’s full guideline set raised tokens from 110K to 166K, 51%, while underperforming curated retrieval.
- AppWorld is the only benchmark validating the reported results, so broader agent evaluations and real-world deployments remain unfinished.
- Cosine similarity does not reliably identify the most useful guidelines, and controlled tests of context-window effects have not been completed.