CrucibleBench finds an LLM judge shifted rankings by six spots
Benjamin Davis and Philip Mims spent $99.59 on 650 persistent text-world runs across 13 models, then treated judge instability as the result.
By Ryan Merket · Published
Primary source: CrucibleBench
Why it matters
Cheap, inspectable worlds can make agent failures legible, while CrucibleBench shows how a benchmark's own judge can become its largest source of ranking instability.
In May, Benjamin Davis and Philip Mims published a CrucibleBench experiment that sent 13 language models into a persistent text dungeon for 650 runs. Their most useful result came from the machinery doing the grading: removing two classifier-dependent scoring dimensions moved models by as many as six leaderboard positions.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.