An LLM workflow re-ran 4,452 economics replication packages and flagged discrepancies in 3,460

Matthew Schwartz, Isaiah Andrews and Jesse M. Shapiro built a multi-agent LLM workflow and ran it on every available replication package for articles published since 2000 in Econometrica, the Quarterly Journal of Economics, the American Economic Review, the Journal of Political Economy and the Review of Economic Studies: 4,452 packages.

For 12.4% of packages the documentation says all the data is restricted, and the workflow attempted no rerun. Across all 4,452, 11.2% reproduced in full and 76.4% did not. A second, adversarial agent tried to overturn each discrepancy, and the authors reran the calculation in the original software where they could license it. Of the calculation mismatches, 35.6% fall within one unit of the last printed digit, which is consistent with rounding, and 24.9% differ in the first significant digit.

The workflow also rewrote code. It cut a calculation’s running time by more than a factor of ten in 496 articles and proposed an extension that keeps the original assumptions in 923.

The authors describe the numbers as current and the work as ongoing, with a public release planned at github.com/MetaEconomics. Schwartz worked as a contractor for Anthropic on the project. The paper cites a working paper by Dmitriy Muravyev that uses an LLM pipeline to reconstruct the main result of 1,328 articles in three finance journals.