Ai2 Study Finds LLM Research Novelty Judges Are Highly Unreliable
Ai2 finds LLM novelty judges highly prompt sensitive, with GPT 5.4 accuracy falling from 93.5% to 40.9% and costly search offering no clear gain in tests.
Summary
On October 27, 2026, Ai2 released Old Ideas, Novel Problems, including a paper, benchmark, data, code and an automated evaluation set pipeline. Six core judges, GPT 5.1, GPT 5.2, GPT 5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, assessed ideas individually and in pairs. ICLR 2026 reviews supplied 154 accepted ideas from each area’s top rated 10% with undisputed novelty praise, plus 145 rejected ideas from the bottom 10% with undisputed originality concerns. An annotator matched the extracted signal in 47 of 50 cases. Human+Generated paired the 154 validated human ideas with 154 basic LLM ideas under weak labels; an expert preferred the human idea in all 30 checked pairs and found none of the generated ideas novel.
Small setup changes produced conflicting verdicts. Simplifying GPT 5.4’s prompt cut Human+Generated accuracy from 93.5% to 40.9%, below chance, and reversed more than half its decisions; the equivalent Human Only change was about seven points, although Claude Opus 4.6 shifted 21 points. Prompt changes altered roughly one quarter to one third of verdicts comparing AI generators. GPT 5.6 Sol, using extra reasoning and live arXiv search, scored 0.78 macro F1, 11 points behind Claude Opus 4.6 at 0.89, while costing roughly ten times more. A simple Claude Opus 4.6 baseline scored 0.854 for $2.59, beating Idea Novelty Checker at 0.642 and $30.66, and Sakana AI’s AI Scientist at 0.772 and $82.05. Ai2 recommends testing judges on realistic ideas across prompts and models, improving literature integration and continually updating benchmarks.
Positives
- Ai2 released the paper, benchmark, data, code and a reusable pipeline for constructing new novelty evaluation sets.
- 47 of 50 review passages received matching novelty signal classifications from an independent annotator.
- All 30 difficult Human+Generated pairs were resolved in favor of the human idea by an expert, who found no generated idea novel.
- Claude Opus 4.6 reached 0.854 macro F1 for $2.59 using low reasoning and majority voting across three calls.
- The reusable pipeline can extend testing to new peer reviews, scientific communities and future novelty judges.
Risks & concerns
- GPT 5.4 accuracy collapsed from 93.5% to 40.9% after a few prompt words changed, reversing more than half its verdicts.
- Claude Opus 4.6 accuracy moved 21 points on Human Only data under the same prompt change.
- Prompt changes altered roughly one quarter to one third of judgments when comparing ideas from two AI generators.
- GPT 5.6 Sol scored 0.78 macro F1 despite live arXiv search and roughly ten times the decision cost.
- Idea Novelty Checker and AI Scientist cost $30.66 and $82.05 while underperforming a $2.59 Claude Opus 4.6 baseline.