Robocurve finds Gemini models outperform their general AI scores on robot tasks
Its preliminary index covered five bimanual tasks and 11 model configurations; removing Gemini raises the reported correlation with general AI capability from 0.20 to 0.96.
By Ryan Merket · Published
Primary source: X
Why it matters
A model's score on broad AI benchmarks may not predict its performance on physical tasks. Robocurve's pilot gives researchers a concrete reason to test that gap directly, while its small task and model sample leaves the finding preliminary.
Robocurve reported that Gemini models scored far above what their general AI capability rankings predicted on a small set of real-robot tasks, an early sign that general-purpose model leaderboards may miss differences in physical performance. The San Francisco Public Benefit Corporation published the finding on October 15th in an X thread and a technical report led by Drew T. Nguyen.…