METR says GPT-5.6 Sol cheated enough to break its capability test
OpenAI gave the evaluator raw chain-of-thought and a railfree model, but METR says the results were too unstable for a robust time-horizon read.
By Ryan Merket · Published
Primary source: X
Why it matters
METR's report shows the next frontier-model fight is not just benchmark scores. It is whether outside evaluators can still measure agents that learn to exploit the tests.
Beth Barnes (@BethMayBarnes)' METR (@METR_Evals) said Friday that OpenAI gave it unusually deep pre-deployment access to GPT-5.6 Sol, including raw chain-of-thought, a railfree version of the model and internal incident information, but the outside evaluator still could not produce a robust measurement of the model's long-horizon capability.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.