Marin debuts Delphi to forecast 25B pretraining runs with 0.2 percent error

In an X thread, Will Held says small-run fits predicted a 25B-param, 600B-token run at ~1e23 FLOPs, matching Paloma macro loss within 0.2% error.

By · Published

Illustration of Marin's Delphi forecasting a 25B-parameter, 600B-token pretraining run at about 1e23 FLOPs and matching Paloma's macro loss within 0.2%

In an X thread, Will Held says small-run fits predicted a 25B-param, 600B-token run at ~1e23 FLOPs, matching Paloma macro loss within 0.2% error.