Artificial Analysis brings final-state scoring to agent workflow benchmarks
Micah Hill-Smith and George Cameron are testing agents against SaaS workflows where the database state matters more than the answer text.
By Ryan Merket ยท Published
Why it matters
Agent startups are being valued on the promise that models can act across business systems. AutomationBench-AA gives buyers and builders a harder metric than a demo: final-state task completion with guardrails.
Micah Hill-Smith (@micahhsmith) and George Cameron (@georgecameron) launched AutomationBench-AA on July 7, 2026, giving Artificial Analysis an independent leaderboard for a question that is quickly becoming central to enterprise AI buying: did the agent actually finish the workflow?
Artificial Analysis announced the launch on X and on LinkedIn, describing AutomationBench-AA as its independent leaderboard for Zapier's AutomationBench. The benchmark tests AI agents on simulated SaaS workflows, then grades the final state of the simulated environment rather than asking another model to judge the agent's answer.
That distinction is the point. A sales ops task does not end when an agent writes a plausible summary. It ends when the correct CRM record has been updated, the right email has been sent, the wrong recipients have been avoided, the stale spreadsheet row has been ignored, and the policy hidden in an inbox has been followed. AutomationBench-AA is Artificial Analysis' attempt to make that kind of work legible in a public model ranking.
Hill-Smith, Artificial Analysis' co-founder and CEO, came to the problem from the builder side rather than the lab side. In a Latent Space interview, the co-founders described Artificial Analysis as beginning in 2023 as a side project while Hill-Smith was building a legal AI assistant and trying to choose the right model for different parts of the system. The original pain was practical: accuracy, speed, and price were all moving, and the public data was too thin to make reliable engineering decisions. McKinsey, where Hill-Smith had previously worked as a business analyst in Sydney from 2020 to 2022, later described him as the co-founder of an AI benchmarking company tracking model development and efficacy.
Cameron, Artificial Analysis' co-founder and chief product officer, framed the company bluntly on Latent Space: "you can't pay us for better results." Hill-Smith said in the same interview that no one pays to appear on the public website. The business, according to the co-founders, comes from enterprise benchmarking insights and private custom benchmarking, which makes the public leaderboard a distribution asset and a trust claim at the same time.
What AutomationBench-AA changes
AutomationBench-AA is built on Zapier's AutomationBench, a benchmark introduced in April 2026 for end-to-end business workflow execution. Zapier says AutomationBench uses 47 real tools across six business functions - Sales, Marketing, Operations, Support, Finance, and HR - and is based on workflow patterns from more than 2 billion monthly tasks across 3.7 million companies.
The underlying AutomationBench paper, posted to arXiv in April 2026, says existing automation benchmarks often miss the combination that matters in real business work: cross-application coordination, autonomous API discovery, and policy adherence. In Zapier's framing, the agent has to find the right endpoints, follow layered business rules, and write the correct data to the right systems.
Artificial Analysis' version uses a private 657-task held-out split from AutomationBench version 1.0, according to its methodology page. The tasks span Finance, HR, Marketing, Operations, Sales, and Support, and run in simulated app environments including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot. Each task runs once in a multi-turn environment with a 50-turn cap. Models use REST API tools and must discover and call the endpoints they need through structured tool calls.
The scoring mechanism is where Artificial Analysis made a choice that changes the leaderboard. Zapier's own AutomationBench leaderboard emphasizes strict full task completion: every scored assertion has to pass. Artificial Analysis instead reports a headline AutomationBench-AA score based on the share of task objectives completed, with a task receiving zero if the model violates any guardrail. Objectives and guardrails are graded through programmatic checks on the final environment state, and Artificial Analysis says AutomationBench-AA does not use a separate LLM judge for grading.
That makes the leaderboard less binary and more diagnostic. A model that completes most objectives while avoiding unsafe side effects gets credit for the work it actually did. A model that completes useful work and breaks a policy gets zero for that task. The methodology gives model builders a denser signal without letting them paper over the failures that would matter in production.
The scores are still low
Artificial Analysis' leaderboard shows Claude Fable 5 with adaptive reasoning, max effort, and Opus fallback at the top of AutomationBench-AA with a 48.6% score. Other configurations are close behind.
Those scores should not be read as agents completing half of real office work. Artificial Analysis' headline metric is objectives completed after guardrail filtering, not strict end-to-end success. Zapier's own leaderboard remains harsher: it lists Claude Fable 5.0 Max at 17.4% task completion and $3.67 per task, followed by Claude Fable 5.0 XHigh at 16.0% and Claude Opus 4.8 XHigh at 15.5%.
The spread between the two leaderboards is useful. It says current frontier agents can often make partial progress through messy workflows, while full correctness remains scarce. That gap matters for enterprises because most internal workflows tolerate some human-in-the-loop recovery during pilots, then become much less forgiving once an agent touches customer data, finance systems, support routing, or HR records.
Zapier's own example makes the failure mode concrete. A task may require an agent to mark the correct Meridian Corp opportunity as won, convert currency from the newest spreadsheet row, identify the right account tier, find a routing policy in email, and notify the executive and support escalation teams while avoiding several wrong recipients. Passing five of six assertions can look impressive during a demo. In the strict metric, it still fails.
Why Artificial Analysis is leaning into agent evals
The launch moves Artificial Analysis deeper into agent evaluation at a time when static model rankings are losing some of their usefulness. The company still publishes leaderboards across model quality, speed, latency, price, context, and other dimensions, but the buying question has shifted as companies move from chatbots and copilots toward agents that act across tools.
Artificial Analysis says on its careers page that it has more than 40 employees, with colleagues from Google, Microsoft, McKinsey, and BCG, and that it is backed by Nat Friedman, Daniel Gross, Andrew Ng, Adam D'Angelo, Clem Delangue, Guillermo Rauch, and Swyx. The company also says it is trusted by hundreds of thousands of users. Those are company-supplied figures; Artificial Analysis has not publicly disclosed revenue, ARR, valuation, seed round size, or total funding.
The market around the company is moving in the same direction. Braintrust announced an $80 million Series B led by ICONIQ on February 17, 2026 to build production AI observability infrastructure. LangChain launched LangSmith Engine in public beta in June 2026, pitching a system that clusters production failures, diagnoses root causes, and proposes fixes and eval coverage. Patronus AI raised a $50 million Series B on June 25, 2026 to build simulated digital environments for stress-testing agents.
Artificial Analysis occupies a different position in that market. Braintrust and LangSmith sit closer to production teams trying to observe, debug, and improve their own applications. Patronus is selling simulated environments and stress tests. Artificial Analysis is trying to become the independent scoreboard that engineers, labs, and buyers can check before they choose which model should run the workflow in the first place.
That is a valuable position if the methodology holds. It is also exposed to the usual benchmark pressures: model providers optimize for public tests, private splits eventually leak signal, and leaderboard metrics can become product marketing before they become operating truth. Artificial Analysis' answer has been to run evaluations independently, disclose methodology, and use controls such as mystery-shopper accounts so providers cannot easily identify benchmark traffic. AutomationBench-AA adds another layer to that strategy by ranking agents on the state they leave behind.
The immediate readout is straightforward: the best models can do meaningful pieces of SaaS work, and even the best models fail enough that full autonomy remains a hard sell for high-risk workflows. Hill-Smith and Cameron are betting that the companies adopting agents will need an outside scorekeeper that measures the difference.