Community participants using software agents attempted to reproduce 2,226 research papers accepted to ICML 2026 during a 19-day challenge organized by Hugging Face. From July 15 to August 2, 2026, 1,221 participants produced 6,816 logbooks reviewing roughly 34 percent of the conference's 6,352 accepted papers.
Hugging Face published the results on August 13, 2026, after an automated judge running the open-weights model GLM-5.2 evaluated 35,908 individual claims. Participants used agents including Claude Code, Codex, Cursor, and OpenResearch's orx to run experiments using $20 in Hugging Face compute credits per person, launching 2,962 cloud jobs in total.
Reproduction Rates
Out of the examined papers, 1,103 papers—or 51 percent—had at least one claim verified by experiments. Independent teams fully reproduced 266 papers with every extracted claim confirmed, while 632 papers were partially reproduced without any falsified claims. In total, participants confirmed 3,978 individual claims across all submissions.
At least one claim was falsified or contested in 496 papers, representing 23 percent of those tested. Participants found that all claims were falsified in 49 papers, while independent reproduction teams reached opposite verdicts on the same claims across 242 papers. Another 502 papers yielded only reduced-scale evidence, and 280 papers were inconclusive due primarily to missing artifacts.
Identified Errors
Hugging Face re-verified 35 formal falsification claims by re-implementing experiments and checking mathematical proofs. On a paper studying learning-augmented paging, re-testing confirmed a proof flaw that increased the algorithm's additive growth term beyond its claimed limits. In a study on attention and Frank-Wolfe optimization, three separate teams discovered mathematical counterexamples that first appeared after 224 steps.
Another audit of a continual learning paper revealed that released code defaulted to forward KL divergence, despite the paper's theoretical framework analyzing reverse KL divergence. In an evaluation of Transformer cache reduction, a participant found that EOS padding tokens accounted for approximately 66 percent of evaluated positions, understating the quality cost of the approach when corrected.
Human Oversight
While coding agents accelerated execution, Hugging Face found that autonomous systems routinely hit technical limitations without human direction. Unsupervised agents fell into local execution loops, misjudged scale-dependent behavior, and occasionally produced incorrect falsifications due to calculation bugs. The most successful reproductions relied on human participants to re-steer agents, challenge assumptions, and conduct subjective evaluations like visual quality checks.
