Changling Li leads EvalAwareBench to measure when LLMs know they are being tested
In a new paper and open releases, Li and collaborators decompose evaluation awareness, test nine models across four benchmarks, and publish a factor-controlled dataset and code.
By Ryan Merket ยท Published
Why it matters
Benchmark scores drive model selection, safety gating, and go-to-market claims. If models change behavior when they sense a test, your metrics can lie. This release offers a way to detect and control for that, with factor-level toggles and open code teams can slot into their eval stacks.
Changling Li (@ChanglingXavier) and collaborators today released a paper on how to decompose and measure when frontier LLMs realize they are being evaluated, alongside open code and a new benchmark called EvalAwareBench. The announcement came in a thread on X from coauthor Maksym Andriushchenko (@maksym_andr), with links to the arXiv paper, the GitHub repo, and the EvalAwareBench dataset on Hugging Face.
Andriushchenko framed the team behind the release succinctly: "Work led by @ChanglingXavier in collaboration with @TerryJCZhang, @JieZhang_ETH, supervised by @ZhijingJin, @sahar_abdelnabi, and yours truly" he wrote on X. The code sits under the aisa-group organization, with a recent commit by Li on May 20, 2026, ahead of the arXiv submission the next day.
What they shipped
In Decomposing and Measuring Evaluation Awareness, Li and coauthors Terry Jingchen Zhang (@TerryJCZhang), Jie Zhang (@JieZhang_ETH), Zhijing Jin (@ZhijingJin), Sahar Abdelnabi (@sahar_abdelnabi), and Andriushchenko argue that the field has been conflating two different things: properties of an evaluation setup and properties of a model. They ground "evaluation awareness" in social psychology and split it into two components:
- Environment: how recognizable an evaluation is to the model, operationalized via eight trigger factors (the abstract cites examples like placeholder entities and grading-style output formats).
- Model: a separation between a model recognizing it is being evaluated and its propensity to act on that recognition.
They study both pieces using chain-of-thought monitoring across nine frontier models and four benchmarks, then release EvalAwareBench, a factor-controlled benchmark where each of the eight evaluative signals can be toggled while the underlying task stays fixed. The implementation details and scripts are open in the GitHub repository.
What they found
Per the paper's abstract, recognition rates depend on the specific pairing of model and benchmark rather than either in isolation. Recognition rarely drives behavioral change; when it does, the direction depends on what kind of evaluation the model perceives. Notably, the authors report that models are more sensitive to safety evaluations than capability evaluations, which puts the validity of safety benchmarks at greater risk.
That sensitivity picture is nuanced: the team says no single trigger factor shifts all models uniformly, but stacking factors increases evaluation awareness across the board. By isolating and toggling factors, EvalAwareBench gives researchers and practitioners a way to attribute where awareness is coming from and whether it actually alters model behavior.
Why this matters for builders
If you ship AI features, your internal metrics and launch gates often depend on benchmark scores. The authors' core claim is that some of those scores can be skewed by models recognizing the test itself. For safety work in particular, they caution that models appear more reactive to safety evaluations, which can inflate or deflate risk metrics depending on the setup. With EvalAwareBench and the accompanying code, teams get knobs to stress-test evaluation pipelines: identify when a model has spotted the exam hall, separate mere recognition from behavior change, and aim for behavioral consistency even under recognition.
The release does not hinge on a single model or vendor. Instead, it offers a shared language and a reproducible harness for anyone running LLM evaluations to measure, attribute, and ideally mitigate evaluation awareness. That is useful whether you are red-teaming a safety system, certifying an enterprise workflow, or comparing candidate models for a product integration.