LLM Novelty Evaluation

Ai2 Study Finds LLM Research Novelty Judges Are Highly Unreliable

Ai2 finds LLM novelty judges highly prompt sensitive, with GPT 5.4 accuracy falling from 93.5% to 40.9% and costly search offering no clear gain in tests.