Google's ScientistTwo starts from a published paper and returns a better method, code and manuscript
Given 107 problems from top machine learning venues, the system advanced 86 and beat the published method by 25.2% on average. AI reviewers did most of the grading.
Jaehyun Nam, Jinsung Yoon, Tomas Pfister and four coauthors, six of the seven at Google Cloud AI Research, posted ScientistTwo on arXiv on 17 September 2026. The division of labor is stated in one line: “the human researcher defines the target scientific challenge, while the AI autonomously orchestrates the path to advance it.”
Every stage has an agent that does the work and a critic that can send it back
| Stage | It does this |
|---|---|
| Find the weakness | A Limitation Extractor lists what the published method gets wrong. A Limitation Verifier sends the list back until it is sufficient. |
| Propose | An Idea Generator writes candidate fixes. A Novelty Checker scores each one. |
| Screen cheaply | Ideas are tried on a subset of the benchmarks first, and weak ones are dropped before the full run. |
| Run the experiments | The surviving idea is tested across the full set of datasets and metrics. |
| Ablate | Components are removed one at a time. An Ablation Critic decides whether the method needs more changes. |
| Draft | An Initial Drafter writes a manuscript in conference format from the results. |
| Review and rebut | A Peer-Reviewer Agent scores the draft on the ICLR scale. Below the threshold, a Rebuttal Planner designs new experiments and a Rebuttal Coding Agent runs them. |
The experiments ran on Gemini 3.6 Flash and Claude Opus 4.8. The review and rebuttal cycle repeats until the reviewer’s score reaches the acceptance threshold, for which the paper gives 8 as an example, or the budget of iterations runs out.
AI reviewers gave most of the grades
ScholarPeer, the reviewer used inside the loop, accepts 91.9% of ScientistTwo’s papers. The authors call that an in-distribution test and add a second reviewer the system never saw during development, the Stanford Agentic Reviewer. It accepts 72.1%, and it accepts no paper from the earlier research agents in the comparison, AI Scientist-v2 and AutoResearchClaw among them.
Under both reviewers the papers outscore the average accepted paper at ICLR 2026 and NeurIPS 2025. The authors state the ceiling themselves: the system “does not yet consistently achieve the caliber of spotlight or oral presentations” and works by local refinements of an existing method.
Humans graded a smaller set. Nine experienced reviewers read 33 of the papers. In paired comparisons with accepted human papers they found overall parity, preferred ScientistTwo on the breadth of its experiments and gave humans a slight advantage in methodological rigor.
Finance publishes the same input and lacks the score
ScientistTwo needs three things from a problem: a published paper, its code and a benchmark number to raise. Finance journals with a replication policy already publish the first two. An LLM workflow has rerun 4,452 economics replication packages and proposed extensions to 923 of the articles.
The third does not transfer. A forecasting method can be 25.2% better on a benchmark. A coefficient in an event study cannot be “better”, and a loop told to raise a Sharpe ratio until a reviewer approves is a specification search with a manuscript attached. The stage that carries over is the first one in the table: an agent that reads a published finance paper and lists what its method gets wrong.
Sources
- arXiv arxiv.org