Original Analysis

When an AI research idea becomes evidence

A novelty rating, a completed experiment and a favorable review support different claims. Studies of AI research show why confidence should follow the stage actually tested.

Published

Four claims that should not travel together

TideReach assessment

An AI research proposal can be original without producing a useful experiment. An experiment can run without supporting the paper written about it. A favorable review adds another judgment, rather than settling every earlier question. The useful way to assess AI research is to ask what each piece of evidence establishes: originality, successful execution, experimental validity or independent scrutiny. [1][2][3][4]

A novelty score answers a limited question

Old Ideas, Novel Problems, revised in October 2026, finds that plausible changes to an LLM novelty judge can produce large verdict shifts on the same idea pairs. Its generated-idea labels are partly assumptions, checked by one expert on thirty selected pairs. That limits claims about absolute accuracy, while leaving disagreement between judge configurations observable. [1]

Context

There is also positive evidence for AI ideation. A 2024 study by Chenglei Si, Diyi Yang and Tatsunori Hashimoto found that NLP experts, reviewing blinded proposals, rated ideas from its AI pipeline as more novel than human-written ideas. Those were proposal judgments. The generation and evaluation designs differ from the later benchmark, so these findings should not be treated as a direct contradiction. [2][1]

TideReach assessment

For a research team, a novelty rating is a reason to investigate a candidate. Treating it as evidence that the proposed method will work adds a conclusion the rating did not test. [1][2]

What changes when people execute the ideas

The 2025 Ideation–Execution Gap study asked forty-three experts to implement randomly assigned NLP prompting proposals, then used blinded review of the resulting work. AI-generated proposals lost more of their earlier ratings after execution. But the final human-versus-AI difference was not statistically significant in two-sided tests when each project's average rating was counted as one observation. One-sided tests treating individual reviews separately produced different significance results. [3]

TideReach assessment

That distinction changes the story the evidence supports. The study shows why promising proposal ratings need an execution test; it does not establish that human ideas universally produce better research. Several reviews can describe one project, so the number and kind of independent observations matter alongside the headline score. Humans also performed the implementations: this was not a comparison of autonomous scientists. [3]

Executed work still needs an audit

The AI Scientist-v2 report describes three workshop submissions, one of which scored above the acceptance threshold; the submissions were withdrawn by prior arrangement. Its authors also report code audits finding substantial training-and-test data overlap in one setup and temperature-scaling code that was implemented but not run. These are the authors' checks, not a TideReach reproduction. [4]

TideReach assessment

The lesson is more specific than whether an AI can write a plausible paper. A result needs a traceable connection between the claimed experiment, the run that occurred, the data used and the conclusion in the manuscript. Execution can establish that a workflow completed while leaving that connection uncertain. [4]

Give each result the confidence it has earned

TideReach assessment

A useful evaluation would keep the stages visible. An originality claim needs a defensible comparison with prior work. An execution claim needs identifiable experiments and artifacts. A validity claim needs appropriate controls, data separation and analysis. Review and replication add scrutiny, with their own scope and limitations. A strong signal at one stage should not silently become a pass at every other stage. [1][2][3][4]

TideReach assessment

This leaves room for useful AI research, including negative results. A well-designed experiment can be valuable when a proposed method fails. Screening more ideas can also help researchers choose where to spend effort, even when most proposals will not survive testing. The relevant question is whether the evidence supports the particular claim being made. [3][4]

Context

These papers examine particular systems, tasks and evaluation designs, including older models. They do not provide a common league table for today's AI researchers, and TideReach has not reproduced their experiments. The practical conclusion is therefore a reporting standard: describe the stage reached, retain the qualifications, and identify the next check needed before making a stronger claim. [1][2][3][4]

Sources and further reading

Numbered references connect each passage to its sources and related TideReach coverage.

  1. arXiv; Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue and Tom Hope Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation Back to text ↑
  2. arXiv; Chenglei Si, Diyi Yang and Tatsunori Hashimoto Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers Back to text ↑
  3. arXiv; Chenglei Si, Tatsunori Hashimoto and Diyi Yang The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas Back to text ↑
  4. arXiv; Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune and David Ha The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search Back to text ↑

Explore more articles