IBM ALTK-Evolve Beats ACE on AppWorld With Up to 85% Fewer Tokens

Summary

On August 11, 2026, IBM Research’s Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Punleuk Oum, G Thomas, Ashwath Vaithinathan Aravindan, Evelyn Duesterwald and Merve Unuvar said ALTK-Evolve and ACE turn an LLM agent’s trajectories into inference-time memory without weight updates or human labels. Both preserve distinct lessons and evidence counts rather than compressing them. ACE incrementally builds, embedding-deduplicates and injects one full playbook at every step through a Generator, Reflector and Curator loop. ALTK-Evolve clusters and support-conservingly merges near-duplicates, records strategy, recovery and optimization guidelines with causal provenance at subtask level, then supplies a high-support core plus cosine or LLM-guided, priority-weighted retrieval, or the full 191-guideline set when a stronger model can use it.

In IBM’s controlled AppWorld test_normal benchmark, 168 pass@1 tasks using the same ReAct code agent and train/dev-mined memory, DeepSeek-V3.2 with ACE’s 106 guidelines scored 80.4 TGC and 73.2 SGC at 634K tokens per task; ALTK-Evolve scored 89.3 and 80.4 at 263K, versus its 79.8 and 64.3 no-memory baseline at 148K. On gpt-oss-120b, ACE scored 54.8 and 35.7 at 777K; ALTK’s roughly 29 selected guidelines scored 56.0 and 37.5 at 116K, versus 39.9 and 21.4 at 110K. ACE led gpt-oss Easy and Medium, but ALTK won Hard 31.8 to 23.8 and aggregate; on DeepSeek, ALTK led Easy, Hard and overall while ACE led Medium. A repeat ALTK gpt-oss run scored 54.8, so IBM calls accuracy tied within noise. Results are single runs; IBM reran ACE because its paper used DeepSeek-V3.1, and differing prompt templates produced DeepSeek no-memory TGC baselines of 79.8 and 72.0. TGC means Task Goal Completion; SGC requires every scenario variant to pass. IBM released the ALTK-Evolve library and technical report, with injection scaling deferred to a next post.