Qwen 3.8-Max and Claude Opus 5 Tests Expose the Real Cost of AI Reasoning

Summary

Alibaba released Qwen 3.8-Max during the week of August 6, 2026, promoting the preview as trailing only Claude Fable 5. However, Alibaba’s launch table showed Qwen leading just one of 12 coding-agent rows. An independent VulcanBench run, apparently using the Preview model, produced a sharply different picture: Qwen’s strongest effort configuration ranked around the middle of the field, while its default configuration finished last. Both sets of results can be valid because they were produced under substantially different operating constraints.

The central factual difference was the available execution time. Alibaba’s coding tests allowed a five-hour timeout, while PaperBench runs could continue for as long as 12 hours. VulcanBench imposed wall-clock limits of 45 to 60 minutes. Depending on the comparison, Alibaba therefore provided roughly five to 16 times as much time. The article’s interpretation is that benchmark rankings cannot be evaluated responsibly without disclosing time, token and reasoning budgets alongside accuracy scores.

Published token prices also provide an incomplete picture. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, was listed at $0.14 per million input tokens and $0.28 per million output tokens. Qwen 3.8-Max cost $2 and $6 respectively, while Kimi K3 was priced at $3 and $15. Yet Artificial Analysis found that DeepSeek-V4-Flash used 210 million output tokens at maximum effort on its Intelligence Index, compared with a 100-million-token class median. Its low rates kept the monetary cost down, but the extra generation could still increase latency or cause a run to hit its budget before producing an answer.

Evidence from other tests shows that greater reasoning effort does not always improve the number of completed tasks. Long-Horizon-Terminal-Bench tested 17 frontier models on 46 tasks with one 90-minute attempt each. Timeouts represented 79% of unresolved runs, compared with 19% in which agents stopped themselves and 3% attributed to harness errors. The authors did not claim that more time would have rescued those runs: timed-out attempts had mean rewards of only 0.10 to 0.35. In VulcanBench’s July 26 report, Claude Opus 5 solved 20 of 23 tasks at its lowest effort but only 18 at high effort. High effort produced just one wrong answer versus three at low effort, yet more runs expired. Even with unlimited time on the two cited regressions, it would only tie the cheapest setting while costing 3.1 times as much.

The recommended metric is therefore total cost per successful task: all spending, including failed or empty attempts, divided by work that passes a defined acceptance check. Supporting examples include GPT-5.4 costing roughly $26 per task in Long-Horizon-Terminal-Bench while passing far fewer tasks than Grok 4.5 at about $11, and Claude Code’s TestEvo-Bench score dropping from 71% to 44% under a tighter cost cap. Operators are advised to record whether failures came from budget exhaustion, verifier rejection or harness errors; compare cost per success at every effort level; and prefer token caps unless latency is an explicit service requirement. One immediate uncertainty is how Qwen 3.8-Max would rank under standardized budgets. The operational warning is clearer: when its effort field is unset, Qwen defaults to its highest reasoning level, which was its costliest and worst-performing configuration in the cited independent test.