ThinkingBox RL Nearly Triples Qwen3.8-27B Agent Reliability
Microsoft and Hugging Face's ThinkingBox raises Qwen3.8-27B success to 60.78%, nearly triples repeatable wins and cuts dependable-task costs by 67.2%.
Summary
On October 22, 2026, Microsoft and Hugging Face released ThinkingBox-Training on GitHub, turning executable checks of final database state into binary reinforcement-learning rewards. After 50 token-mean GRPO steps, Qwen3.8-27B’s pass@1 on 507 unseen ThinkingBox-Bench tasks, each run 20 times for 10,140 attempts, rose from 5,242 successes, 51.70%, to 6,163, 60.78%. Tasks passed on all 20 attempts climbed from 38, 7.50%, to 107, 21.10%, while pass@20 coverage fell from 453, 89.35%, to 446, 87.97%.
Training used 187 disjoint tasks selected by offline policy-relative difficulty. Repeated base-policy probes retained sometimes-solved tasks likely to yield mixed rewards, while nearly always solved and rarely solved tasks were dropped. Full-parameter training ran 144 episodes per update across 24 H100s, with GPT-5 Chat simulating users and assisting a minority of rubric checks. Pass@1 and 20/20 reliability improved across all five domains, but pass@20 fell in three. Retail gained 24.5 points in 20/20 performance, while auto insurance lost 7.0 coverage points. On a hotel task, unauthorized changes occurred in 15 of 20 base attempts and none after training, although all expected records matched in 19.
Estimated evaluation cost fell from $925.80 to $855.58, cost per successful attempt from $0.177 to $0.139, and cost per dependable task from $24.36 to $8.00, excluding training and probing. The model’s 107 fully repeatable tasks beat Kimi-K3’s 68, GPT-5.6 Sol’s 82 and Claude Sonnet 4.6’s 102, but trailed GPT-5.4’s 128, GPT-6 Astra’s 231 and Claude Opus 5.5’s 241. Transfer remains limited because both sets share ThinkingBox and similar task styles. The MIT-licensed code is public, but the private 187-task pool is not.
Positives
- Qwen3.8-27B pass@1 increased 9.08 points, from 51.70% to 60.78%, after 50 GRPO steps.
- Tasks completed correctly in all 20 attempts rose from 38 to 107, a 2.8-fold increase.
- Cost per dependable task dropped 67.2%, from $24.36 to $8.00.
- Pass@1 and observed 20/20 performance improved across retail, auto insurance, travel, neobank and consulting.
- Unauthorized hotel-booking changes fell from 15 of 20 attempts before training to zero afterward.
- ThinkingBox-Training publicly releases the selection, rollout, reward and Verl integration code under the MIT license.
Risks & concerns
- Pass@20 coverage declined from 453 tasks, 89.35%, to 446 tasks, 87.97%.
- Auto insurance recorded the largest breadth loss, with pass@20 falling 7.0 percentage points.
- The $855.58 serving estimate excludes training and policy-probe costs, so it is not a complete deployment bill.
- Training and evaluation shared the ThinkingBox framework, simulated-user machinery and task style, limiting the transfer claim.
- The private 187-task training pool is unavailable, preventing exact reproduction with the public sample list.
- Qwen3.8-27B remained behind Claude Opus 5.5, GPT-6 Astra and GPT-5.4 on fully repeatable benchmark tasks.