Artificial Analysis starts counting when coding agents refuse the job
Micah Hill-Smith and George Cameron's index now separates blocked attempts from agents that recover through a fallback.
By RuntimeWire Staff · Published
Primary source: Artificial Analysis on X
Why it matters
Coding-agent performance depends on provider policies as well as model capability. By separating blocked attempts from successful fallbacks, Artificial Analysis gives developers a clearer view of whether an agent can complete legitimate security and terminal work when safeguards intervene.
Artificial Analysis, co-founded by Micah Hill-Smith (@_micah_h) and George Cameron, added safety-refusal reporting to its Coding Agent Index on September 18th, exposing how often an AI provider or model declines benchmark work on safety grounds. Artificial Analysis announced the addition on X alongside version 1.5 of its coding-agent benchmark.
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.