Sherpa Labs launches AI benchmarks built around finance deals and production code
Founder Justin Jiang says the benchmarks expose gaps in long-horizon work; the finance test recorded a 3.6% overall pass rate across 3,750 runs.
By Ryan Merket · Published
Primary source: X
Why it matters
Sherpa Labs is selling the infrastructure to train and measure agents on professional workflows, where a plausible answer can still fail when formulas, files or software changes do not tie together. Its early scores illustrate the gap the company hopes labs will pay it to close.
On October 20th, Justin Jiang (@jostinjiang) introduced Sherpa Labs' finance and coding benchmarks for AI agents, tests built around completing professional work rather than answering isolated questions. Jiang says the company is building reinforcement-learning environments for finance, coding and quantitative research, with the first two benchmarks reporting low pass rates on tasks drawn from private-equity deal work and a production software repository.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.