The development

What happened

Ai2 released a benchmark, paper, data and code for evaluating language models as judges of research-idea novelty. The benchmark uses review-derived ideas from ICLR 2026 and a separate comparison with generated ideas. Researchers report that a prompt change produced a large accuracy swing for GPT-5.4 in the generated-idea setting.

Key facts

  • The review-derived pools contain 154 higher-novelty and 145 lower-novelty human ideas from ICLR 2026.

  • A second setting pairs the 154 higher-novelty human ideas with generated ideas using weak labels, not individually verified novelty verdicts.

  • Researchers report that GPT-5.4 accuracy changed from 93.5% to 40.9% after a prompt change in the generated-idea setting.

  • The researchers report that retrieval and extra reasoning did not consistently improve results.

The context

Why it matters

The reported prompt sensitivity raises questions about relying on model novelty judgments without testing their robustness. The weak labels and specific evaluation settings limit what the results establish; they do not prove that all generated research ideas lack novelty.