Vals-Smith turns merged pull requests into private coding-agent benchmarks

Why it matters

Coding-agent procurement is moving from public leaderboard scores toward evidence drawn from a buyer's own repository. Vals-Smith gives Vals AI a route from benchmark publisher to infrastructure embedded in model selection and deployment.

Vals AI (@ValsAI) founders Rayan Krishnan (@RayanKrishnan) and Langston Nashold (@langstonnashold) launched Vals-Smith on July 22nd, a tool that turns a GitHub repository's merged pull requests into a private benchmark for testing coding models and agents.

https://x.com/ValsAI/status/2079976484736037316

The product addresses a purchasing problem created by the expanding market for AI coding tools: a model that leads a public leaderboard may perform differently inside a specific repository, where languages, dependencies, test coverage and engineering conventions vary. In its launch thread, Vals AI described Vals-Smith as a way to measure the percentage of real tasks a model can complete on a customer's own code.

Krishnan and Nashold built Vals AI around the argument that generic and vendor-reported benchmark results are weak proxies for production performance. Both founders studied computer science at Stanford. Krishnan's background includes work at the Stanford Artificial Intelligence Laboratory, Palantir and Microsoft. Nashold previously worked at Hudson River Trading and several large technology companies. In an account of Vals AI's early days, Nashold wrote that he had been sleeping on the floor of Krishnan's Stanford dorm room after a pivot, when Vals AI had no customers or revenue.

Vals-Smith takes that founding thesis into software teams' repositories, giving Vals AI a product tied directly to model selection rather than a public leaderboard that engineers can read for free.

How Vals-Smith builds a benchmark

Users connect a public or private GitHub repository, then choose the models, coding agents and execution harnesses they want to test. Vals AI says Vals-Smith maps the repository's structure, languages and test surface while keeping customer data private. The public page directs prospective users to a waitlist and does not list pricing or immediate self-service access.

Vals-Smith searches merged pull requests for engineering changes that can be reproduced and tested. Each accepted pull request becomes a task containing an issue statement, a starting version of the repository and hidden tests. Vals AI says it rejects candidates that depend on brittle infrastructure or have unclear expected behavior.

Before a task enters the benchmark, its hidden tests must fail against the repository before the original change and pass after the merged fix. During an evaluation, the model receives the issue and the earlier repository state without access to the original developer's patch. Vals-Smith counts a task as resolved when the hidden tests pass and the change does not break existing behavior.

That design follows the execution-based approach popularized by SWE-bench, which built software-engineering tasks from GitHub issues and associated pull requests. Vals-Smith's commercial bet is customization: each engineering team can generate a benchmark from the work its own developers have already completed, including repositories that cannot be placed in a public dataset.

Public repositories show how results can diverge

Vals AI has published sample Vals-Smith results for several open-source projects. A Linux kernel benchmark, updated July 21st, contains 85 tasks tested across 15 models. Vals AI reported that GPT-5.6 Sol completed 63 tasks, while the next two models completed 58 each. The same page tracks cost, latency, token consumption and task-level failures, details that can matter as much as the aggregate pass rate when a team is choosing a production coding agent.

Other public examples on the Vals-Smith page cover Next.js, Bun, Apache Spark, JAX and SGLang. The rankings are not uniform across those repositories. Different models lead on different projects, which supports Vals AI's central sales argument: repository-specific results can produce a different buying decision from a general coding leaderboard.

The launch also extends Vals AI beyond its established business of publishing domain benchmarks across coding, finance, healthcare, law and mathematics. Vals AI already sells evaluation infrastructure for testing copilots, retrieval systems and other language-model applications. Vals-Smith packages one technically demanding evaluation workflow into a narrower product aimed at teams deciding which coding model or agent should touch their code.

Access remains the immediate constraint. The public release is a waitlist, and Vals AI has not published the number of repositories, pull requests or supported agent harnesses required to produce a useful private benchmark. The value of Vals-Smith will depend on whether it can generate enough valid tasks from ordinary repositories without requiring teams to rebuild tests or evaluation environments by hand.