ALTK-Evolve Puts GPT-4.1 Atop τ²-Bench With Learned Memory
ALTK-Evolve lifts GPT-4.1 to 67.5% Pass^4 in τ²-bench retail with learned guidelines, beating top frontier and fine-tuned systems without weight updates.
Summary
On October 20, 2026, IBM Research detailed ALTK-Evolve, which has GPT-4.1 distill successful and failed training trajectories into prompt rules for authentication, confirmations, returns and tool use. The same model extracts and applies them without weight updates, human annotation or test-set leakage. Tests used Sierra's stock τ²-bench ReAct scaffold, GPT-5.5 simulated customers, four trials per task, 40 held-out retail tasks and 20 airline tasks. Lossless memory kept 292 retail and 101 airline rules, versus high-support cores of 31 and 14. GPT-4.1 retail accuracy rose from 78.1% to 82.5%, while Pass^4, success in all four trials, rose from 57.5% to 67.5%. Airline lossless accuracy rose from 57.5% to 65.0%; its Pass^4 reached 40.0%, while the 14-rule core led reliability at 45.0%, up from 35.0%.
More guidance raised per-conversation costs from $0.084 to $0.139 in retail and $0.081 to $0.107 in airline, but matched successful runs shortened from 10.59 to 10.16 and 9.15 to 8.82 assistant turns; prompt caching can reclaim repeated input expense. With GPT-5.5 as agent and extractor, 103-rule sup2 and 146-rule lossless prompts lifted airline Pass^4 from 55.0% to 70.0%; lossless accuracy rose from 72.5% to 82.5% and reproduced exactly across reruns. On the October 12, 2026 full-base leaderboard, GPT-4.1 plus memory ranked first in retail at 67.5%, beating fine-tuned RAFT-30B-A3B's 61.4% by 6.1 points. It ranked eighth in airline at 44%, beating non-reasoning GPT-5.2 and nearly matching Claude Sonnet 4.5, while GPT-5.5 ranked first at 82.0%, above frontier entries at 66% to 72%.
Positives
- GPT-4.1 reached 67.5% retail Pass^4, ranking first and beating fine-tuned RAFT-30B-A3B by 6.1 percentage points.
- GPT-5.5 reached 82.0% airline Pass^4, exceeding frontier leaderboard entries scoring between 66% and 72%.
- Retail lossless guidance raised held-out accuracy from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5%.
- GPT-5.5 airline guidance improved held-out accuracy by 10 points and Pass^4 by 15 points without weight updates.
- Lossless guidance shortened matched successful conversations by 0.43 retail turns and 0.33 airline turns.
Risks & concerns
- Retail's 292-rule prompt raised cost per conversation by 65%, from $0.084 to $0.139.
- Airline's lossless prompt delivered 40% Pass^4, below the compact 14-rule core's 45%, showing that more memory can reduce reliability.
- GPT-4.1 reached only eighth place in airline at 44% Pass^4 despite its learned guidance.
- Guideline selection required domain-specific tuning because retail favored the full rule set while airline favored a smaller high-support core.
- Larger guideline blocks increase input tokens on every model call unless prompt caching offsets the repeated expense.