Ai2 releases a benchmark showing research novelty judges swing with prompts
Researcher Noy Sternlicht and co-authors found one prompt change pushed GPT-5.4 from 93.5% accuracy to 40.9% on AI-generated idea comparisons.
By Ryan Merket · Published
Primary source: Hugging Face Newsroom
Why it matters
AI-for-science systems may use language models to rank ideas as original. Ai2's results show that small prompt changes can reverse those judgments, so novelty scores need testing across setups before they guide research.
A small prompt change sent GPT-5.4's accuracy at choosing the more novel research idea from 93.5% to 40.9%, according to a new study by Ai2 researcher Noy Sternlicht and five co-authors. The result puts a weak link in AI-assisted science under scrutiny: systems asked to generate research ideas may also rely on another language model to certify that those ideas are original.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.