IBM ALTK-Evolve Halves GPT-4.1 AI Agent Consistency Gap
IBM's ALTK-Evolve raises GPT-4.1's five-run AppWorld consistency from 53% to 69%, showing why average AI agent accuracy can conceal repeated failures.
Summary
On September 15, 2026, IBM Research introduced consistency guidelines for ALTK-Evolve after a temperature-zero ReAct agent using GPT-4.1 exposed repeatability failures on 168 AppWorld test_normal tasks. Across five runs, Mean@5 was 77.4%, but Pass^5, requiring success in every run, was 53.0%, creating a 24.4 percentage point consistency gap that reached 30 points on hard tasks. Pass@k instead requires only one successful attempt. Near-tied token probabilities can flip API, argument and retry decisions because of platform perturbations, even with greedy decoding or fixed seeds, and small differences compound across long trajectories.
The black-box Consistency Analyzer identifies unstable decisions from one recorded trace, requiring no ground truth, logits, tool calls, environment interactions or end-to-end replay. It makes one offline model call per decision step, sampling five completions by default, then ALTK-Evolve converts flagged steps into reusable guidelines. Using one baseline trajectory per task and five fresh runs, Pass^5 rose 16 points to 69.0%, Mean@5 increased to 81.0%, and the gap fell to 12.0%, with nearly one-third of previously inconsistent tasks becoming fully consistent. Medium tasks gained 22.9 points, hard tasks 14.3 and easy tasks 12.2. Related tasks gained 13.0 points. With gpt-oss-120b, same-task Pass^5 rose from 10.1% to 16.1%, while related tasks gained 8.7 points. The open-source GitHub toolkit now includes the analyzer and guideline generator, with full methodology on arXiv.
Positives
- Pass^5 increased from 53.0% to 69.0% while Mean@5 also rose from 77.4% to 81.0%.
- The consistency gap fell from 24.4 to 12.0 percentage points without sacrificing average accuracy.
- Medium tasks gained 22.9 points, hard tasks 14.3 points and easy tasks 12.2 points in Pass^5.
- Related AppWorld tasks gained 13.0 points, showing guidelines transferred beyond the trajectories that generated them.
- gpt-oss-120b related-task performance gained 8.7 points, exceeding its 6.0-point same-task improvement.
- One trace and offline resampling eliminate the need for graders, live environment access or full task replay.
Risks & concerns
- GPT-4.1 passed every repeated run on only 53.0% of tasks despite posting 77.4% Mean@5 accuracy.
- Hard AppWorld tasks produced a consistency gap reaching 30 percentage points.
- Temperature-zero decoding and fixed seeds cannot prevent hosted model probability shifts from flipping near-tied decisions.
- gpt-oss-120b remained fully consistent on only 16.1% of same tasks after improving from 10.1%.
- Most benchmark averages conceal whether an agent can repeat a successful result on an unchanged request.
- Each analyzed decision requires an additional offline model call, although it avoids new tool calls and environment interactions.