The AI That Designs Experiments and Writes Research Papers

Imagine a laboratory where the primary investigator never sleeps, never stops revising hypotheses, and writes its own methods section. It sounds like science fiction, but a Japanese startup called Sakana AI has built a system that moves strikingly close to that scenario.

Introduced in 2025, AI Scientist-v2 is an autonomous research engine aimed at machine-learning problems. Feed it a broad topic and it will propose hypotheses, design experiments, generate and run the code, analyze results, produce charts, and draft a paper in the familiar academic format. In a live test, Sakana submitted three fully machine-produced manuscripts to a workshop at ICLR 2025; one earned an average reviewer score of 6.33 out of 10—above the workshop’s usual threshold—and was pulled afterward by prior agreement rather than published.

How does it do all this? The system begins with a loose prompt and spins out multiple candidate ideas. Instead of chasing a single line, AI Scientist-v2 uses a tree-search approach: branches sprout, some are pruned, others are pursued. A supervisory agent guides which experiments deserve more compute and which should be abandoned. The result is not linear trial-and-error but a branching exploration that mirrors how a curious researcher might pivot when an experiment fails.

That branching behavior matters. If an early test returns disappointing results, the system doesn’t blow past the failure. It switches routes, tweaks protocols, or tries a new hypothesis. Short loops of progress, abrupt pivots, and occasional dead ends—these are baked into the engine. It writes the scripts needed, executes them, inspects the data, and converts outcomes into visualizations and narrative. The whole cycle, end to end, can be automated.

For the ICLR trial, organizers were informed that some workshop submissions could be machine-generated, but they didn’t know which ones. The accepted machine-written paper focused on techniques to improve generalization in neural networks. Although the workshop reviewers gave it a score above the usual cutoff, the paper was not taken to full conference publication; the authors and hosts had agreed ahead of time to withdraw accepted items for this experiment.

That single modest success, however, sits beside clear limits. Only one of the three submissions reached the acceptance threshold. Sakana’s own researchers judged none of the pieces ready for the main conference. And v2 is not an unambiguous upgrade over its predecessor: when problems are narrowly defined and templates exist, the older version sometimes performs better. The new system is tailored for exploration of open-ended problems—and with exploration comes a higher failure rate.

Practical risks complicate the picture further. When an AI writes and runs code automatically, it may invoke unintended commands or create security hazards. For that reason the team recommends running experiments inside isolated containers such as Docker. AI Scientist-v2 is open-source on GitHub, but it requires substantial hardware—NVIDIA GPUs and access to large language models—so it’s not yet something most labs can deploy in a spare afternoon.

Automating experimental design and paper drafting is an impressive technical milestone—but it is not a substitute for reproducible, peer-reviewed science.

So where does this leave us? AI can now assist far beyond literature search and draft writing; it can suggest and test ideas with minimal human prompting. Still, automated output and trustworthy discovery are different beasts. Reproducibility, domain expertise, and skeptical peer review remain the cornerstones of reliable research—no matter how clever the algorithm. What will matter next is how human scientists and these systems team up, and whether that partnership can raise the bar for what counts as credible science.