Marin debuts Delphi to forecast 25B pretraining runs with 0.2 percent error
In an X thread, Will Held says small-run fits predicted a 25B-param, 600B-token run at ~1e23 FLOPs, matching Paloma macro loss within 0.2% error.
By Ryan Merket · Published

In an X thread, Will Held says small-run fits predicted a 25B-param, 600B-token run at ~1e23 FLOPs, matching Paloma macro loss within 0.2% error.