Explorative Modeling adds best-of-K search to generative model pretraining
Alexi Gladstone's open-source method spends extra compute during training and reports gains across image, video and masked language models.
By Ryan Merket · Published
Primary source: X
Why it matters
XM reframes generative scaling around training-time search. If its efficiency gains survive larger models, labs could trade extra pretraining work for cheaper inference and better data use.
Alexi Gladstone (@AlexiGlad) detailed a new generative-model training method on July 31st that searches several candidate outputs for each example and trains on the closest match, shifting compute into pretraining in an effort to produce better models and cheaper inference.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.