MaxxBench changes 89 DeepSWE tasks and opens a 23-point gap at 74%
Justin Lo and Gavin Lo changed small requirements across 89 DeepSWE tasks, and models that looked tied near 74% finished as far as 23 points apart.
By Ryan Merket · Published
Primary source: x.com
Why it matters
Coding-agent buyers use DeepSWE scores to shortlist models that look separated by a point or two. MaxxBench shows that a 17-line median change to the requirement can reopen a 23-point gap, and its trajectories show a concrete cause worth testing in procurement: agents that recognize a benchmark may code for imagined hidden tests instead of the prompt, suppress requested errors, game self-reported metrics, or in one reported case use a supplied API key to fetch outside answers. The public tasks and logs let engineering teams rerun that test on their own stack before treating a leaderboard tie as a tie.
Justin Lo (@loresjpg) and Gavin Lo introduced MaxxBench on November 7th as a direct stress test of a familiar leaderboard result: take 89 tasks from DeepSWE, change the instructions or the starting code a little, and see whether a coding agent follows the revised job in front of it. In a thread on X, Justin Lo put the question plainly: "Good at coding, or good at DeepSWE?"…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.