Replit's ViBench 2.0 gives top AI models a 49%-65% whole-product pass rate
The benchmark tests agents building 17 products from scratch, adding features in sequence and checking the finished apps in a browser.
By Ryan Merket · Published
Primary source: replit.com
Why it matters
Whole-product tests expose failures that feature-by-feature coding benchmarks can miss. ViBench's public harness gives teams a way to reproduce Replit's results, while its vendor-built test suite and partial holdout set are important limits on how broadly to apply the scores.
Replit has released ViBench 2.0, a benchmark that measures whether AI agents can build complete software products feature by feature. In Fireworks AI's runs, the top three models passed every test plan in 49% to 65% of builds, according to Replit's announcement and its methodology post.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.