Supabase launches open benchmark for AI coding agents building backends

Why it matters

AI agents now influence which databases and backend tools developers adopt. Supabase Evals gives the platform a way to find and fix agent failures before they become lost projects.

Supabase, founded by Paul Copplestone (@kiwicopple) and Ant Wilson (@AntWilson), launched an open-source benchmark on July 31st for measuring how well AI coding agents build and repair applications on its Postgres platform.

The release formalizes a shift already reshaping Supabase's business. Copplestone started Supabase after running into Firebase and Firestore constraints while building an earlier product, leading him and Wilson to pursue a Postgres-based alternative. Six years later, AI agents have become a significant source of demand for that infrastructure. Supabase said in June that AI tools launch more than 60% of its new databases.

Supabase Evals, developed by Supabase engineer Matt Rossman (@the_ross_man), runs agents including Claude Code, Codex and OpenCode against tasks drawn from support tickets, bug reports and GitHub issues. Examples include building a database schema, debugging a failed Edge Function and repairing a broken row-level security policy.

Each scenario operates against an actual Supabase environment rather than a static set of coding questions, according to Supabase's launch post. Agents use Supabase's Model Context Protocol server or command-line interface, and their work is graded through deterministic checks alongside an LLM judge for tasks requiring semantic assessment.

Turning agent failures into product work

The benchmark serves two purposes inside Supabase. A public suite compares agents across product areas and stages of application development. A separate regression suite tracks specific failure modes daily, allowing Supabase engineers to test whether changes to documentation, agent instructions or products correct recurring mistakes.

The Apache-2.0 licensed repository contains the prompts, scorers, starting project states and experiment configurations. Developers can run individual scenarios or execute benchmark suites across multiple agents. Local-stack tests place the agent inside a Docker sandbox with the real Supabase CLI, while tool-based tests expose the platform through Supabase's MCP interface.

Supabase also gives each agent a retry after an initial failure before assigning a grade. That choice reduces failures caused by a single bad attempt, though it means the scores measure an agent's ability to recover as well as its first-pass accuracy. The inclusion of LLM-based grading adds another judgment layer that users should consider when comparing results.

Early runs exposed problems that conventional code benchmarks would miss. Supabase found agents writing migrations by hand when an existing project used declarative schemas, and choosing older authentication patterns instead of a newer Supabase library for secure Edge Functions. Supabase responded by revising its agent guidance and publishing clearer documentation.

Supabase also found that access to its agent skills produced uneven gains. In the build-stage snapshot published with the launch, Supabase said Claude Opus 5 and Kimi K3 passed every scenario without a Supabase skill loaded. Claude Sonnet 5 rose from 78% to 100% after receiving the skill, while GPT-5.4 mini moved from 78% to 89%.

Documentation behavior varied sharply. Supabase reported that Codex-based agents consulted its documentation more frequently than Claude Code agents. Codex running GPT-5.6 read about eight pages per scenario, compared with roughly two for Claude Code, while Claude Code accessed the docs in fewer than 40% of scenarios even when skills were available. Supabase describes those results as a snapshot because agent runtimes and models change rapidly.

Supabase is measuring its new distribution channel

The benchmark arrives less than two months after Supabase raised a $500 million Series F at a $10 billion pre-money valuation. GIC led the June 4th round, with existing investors participating, Stripe making a second investment, and Georgian and Salesforce Ventures joining as new backers.

Supabase said database launches had increased 600% over the preceding year and that nearly 10 million developers used its products as of the Series F announcement. Those figures are self-reported. Y Combinator, which backed Supabase in its Summer 2020 batch, lists 120 employees and describes Supabase as a fully remote developer platform built around isolated Postgres clusters, authentication, row-level security, APIs and realtime data.

Evals gives Supabase a feedback system for a distribution channel it does not directly control. When an agent chooses an outdated library, misses a security feature or fails to read the documentation, the failure can prevent a developer from adopting Supabase even if the underlying service works as designed. Publishing the benchmark lets Supabase identify those failures, revise the interfaces agents consume and check whether the fixes survive later model updates.

The scores remain specific to tasks and environments designed by Supabase. They do not establish which coding agent performs best across software development, and Supabase has an incentive to improve how every tested agent uses its products. For developers building on Supabase, however, the scenarios provide a concrete record of whether an agent can handle database security, deployment and debugging work that extends beyond generating application code.