Earthmover and Hugging Face tackle the slow part of fast weather AI

Why it matters

Open weather models are becoming easier to obtain, shifting the engineering burden toward data access and validation. Earthmover is positioning its array infrastructure as the layer every model run still needs.

Ryan Abernathey (@rabernat) and Joe Hamman, the climate scientists who founded Earthmover, published a workflow with Hugging Face on September 8th that makes open-weight AI weather models easier to initialize, run and evaluate.

The joint tutorial connects models hosted on Hugging Face with analysis-ready weather data from Earthmover. Its main example pulls ECMWF ERA5 data into Microsoft's Aurora, generates a four-step, 24-hour forecast and compares the output against ERA5 reanalysis. A separate interactive demo lets users run available models and inspect how long each stage takes.

The collaboration puts Abernathey and Hamman back on the problem that led them to start Earthmover in 2022. Abernathey earned a PhD from MIT, spent about a decade teaching and researching ocean circulation at Columbia University, and resigned from a tenured position to build Earthmover. Hamman trained as a civil and environmental engineer at the University of Washington, worked as a scientist at the National Center for Atmospheric Research and co-founded CarbonPlan before joining Abernathey.

Both founders had worked on Pangeo and Xarray, open-source projects designed for multidimensional scientific data. Their founding argument was straightforward: mainstream cloud databases organize information around tables, while weather and climate systems produce arrays spanning time, altitude, latitude, longitude and dozens of atmospheric variables.

The GPU is waiting for the weather

Open weights have removed one barrier to experimenting with AI forecasting. ECMWF publishes AIFS weights, Microsoft distributes Aurora under an MIT license, and Google DeepMind has released WeatherNext 2. The surrounding data pipeline remains heavy.

Earthmover and Hugging Face estimate in the tutorial that a typical forecast requires about 1GB of initial conditions. Backtesting across a year requires roughly 360GB of disk space before saving any model output. Inference may finish within seconds on a compatible GPU, leaving expensive hardware idle while data is downloaded and prepared.

The new workflow streams selected ERA5 variables from Earthmover rather than downloading complete archives. Earthmover stores the data using Zarr, which divides multidimensional arrays into independently readable chunks, and Icechunk, which adds transactions, snapshots and concurrent writes. A user can request specific variables, pressure levels and timestamps without pulling every byte in the underlying dataset.

The tutorial says its demo can produce a 24-hour forecast in under about 30 seconds end to end. Running Aurora locally requires no GPU, although each six-hour step takes two to three minutes on a CPU versus seconds on a GPU. Users without suitable hardware can send the same script to Hugging Face Jobs and store the result in a Hugging Face bucket or compatible cloud storage.

Hardware compatibility still matters. AIFS, for example, depends on FlashAttention and therefore on supported GPU architectures. Hugging Face handles that part of the workflow by providing selectable compute, while Earthmover supplies the model inputs and historical data needed for validation.

Earthmover finds its distribution layer

Abernathey originally described Earthmover as a modern data stack for scientific arrays. The sharper commercial wedge emerged around weather, where forecasts and observations change constantly and customers cannot treat ingestion as a one-time job.

Earthmover opened its Data Marketplace in January 2026 with datasets from ECMWF, NOAA and independent weather providers. Earthmover says the marketplace now contains more than 60 petabytes of analysis-ready weather, climate and Earth-observation data. Free ERA5 access in the tutorial covers records from 1940 through data that lag real time by several months.

That focus followed an earlier shift toward frequently updated weather and geospatial workloads. Earthmover had more than 10 paying customers when it raised a $7.2M seed round in September 2025, according to Abernathey. Lowercarbon Capital led the round, with Costanoa Ventures and Preston-Werner Ventures participating. Earthmover previously raised a $1.7M pre-seed led by Costanoa Ventures.

The Hugging Face collaboration gives Earthmover a direct route to researchers and developers already testing open models. A developer can encounter Aurora or AIFS on the Hub, run the demo, and create an Earthmover account when the model needs initial conditions. Hugging Face gets another workload for its Jobs and storage products. Earthmover gets a practical demonstration of why access to model weights does not eliminate the need for specialized data infrastructure.

A pipeline demo, rather than a forecast benchmark

The September 8th release concerns operations. Earthmover and Hugging Face introduced no new weather model, and the walkthrough does not establish that Aurora, AIFS or WeatherNext 2 is the most accurate or economical option for a production forecasting service.

The Aurora example does include a useful validation step. It compares forecast temperatures with ERA5 at matching timestamps and calculates a latitude-weighted root mean square error. That shows developers how to test an output instead of accepting a generated weather map at face value. A production decision would still require longer backtests, operational data and comparisons across models, regions and forecast horizons.

Hugging Face says it is working to integrate weather models, including WeatherNext 2, into its Transformers library. If that work reduces model-specific dependencies, data access becomes a larger share of the remaining engineering burden. That is the market Abernathey and Hamman have spent years preparing to serve.

Earthmover's bet is that open weather AI will produce many model runners and model variants, each repeatedly consuming the same enormous archives. The models may take seconds. The durable business sits in keeping the data ready every time they run.