Epoch AI audits 15 benchmarks, finds nine flawed
The first batch flags defects in SWE-bench Verified, Humanity's Last Exam and other tests used to compare frontier models.
By RuntimeWire Staff · Published
Primary source: Epoch AI
Why it matters
AI leaderboards guide product claims, deployment decisions and policy. Epoch AI's audits show that some headline scores reflect broken graders and ambiguous tasks rather than model capability.
Epoch AI, the research institute co-founded and led by Jaime Sevilla (@Jsevillamol), launched Benchmark Reviews on September 17th and classified nine of its first 15 reviewed benchmark versions as Flawed. Four received Verified labels, while Epoch AI lacked enough information to judge the remaining two.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.