Edge0 open-sources a 35B model runtime that streams weights from storage

Why it matters

Edge0 shows how sparse models can trade storage traffic for sharply lower memory use, but its most consequential claim - a usable 35B iPhone model - is not yet reproducible from the released code.

Samuel Zeng (@SamuelZengML) open-sourced Edge0 on September 10th, releasing two language models and an inference framework designed to run large mixture-of-experts models without loading their full weight sets into memory.

https://x.com/SamuelZengML/status/2097861839287927139

The headline demonstration is a 35-billion-parameter model running locally on an iPhone. Zeng says the demo uses 1-2.5 GB of peak memory, with no cloud connection or remote GPU. Edge0 also showed the model generating game logic, structured 3D scenes and interactive web experiences on-device.

That iPhone result remains a demonstration rather than the reproducible baseline included with Thursday's release. The Edge0 repository currently lists macOS on Apple Silicon as its supported platform, with an MLX backend tested on M1 through M4 hardware. The published 35B benchmark reports 2.9 GB of peak active memory on a Mac mini with an M4 Pro and 24 GB of unified memory. Edge0 has not published an iOS build or an iPhone benchmark procedure in the repository.

Zeng, whose full name is Xiaodong Zeng, has spent much of his career working on natural-language systems. He led machine-translation work at Alibaba after earning a master's degree from the University of Macau, where he worked on Chinese-Portuguese-English translation. MIT Technology Review named him to its 2017 Innovators Under 35 China list for work that included Alibaba's translation systems and Ant's TaoCafe retail project. More recently, he has been building Audio8, an early-stage speech-model project focused on cloud and on-device voice systems.

What Edge0 released

Edge0's first release includes a 35B model based on Qwen3.5-MoE and an 8B model based on InclusionAI's Ling 3.0 Tiny. Both checkpoints use 4-bit quantization and are distributed with the adapters required by Edge0's runtime. The code and weights carry Apache 2.0 licenses.

The 35B checkpoint occupies roughly 23 GB of storage, according to Edge0's repository, while the 8B release requires about 4.2 GB. Edge0 avoids holding those full checkpoints in active memory by exploiting the sparse structure of mixture-of-experts models. For each generated token, the model activates a small subset of its available experts. Edge0 keeps the full pool on storage and moves the selected weights into memory as needed.

Storage access can erase the advantage if inference repeatedly stalls while waiting for the next expert. Edge0 trains a small "prerouter" network to predict which experts the next step will need, allowing the runtime to begin loading them while the current computation is still running. Frequently selected experts remain cached, while less-used weights stay on disk.

Edge0 also ships what it calls Recover-LoRA, a set of adapters trained to recover quality lost when the base models are reduced to 4-bit precision. In Edge0's own OpenCompass testing, the 35B pipeline averaged 79.2 across five benchmarks, compared with 83.2 for the full-precision base model. The 8B version averaged 69.9, compared with 72.7 for its base. Those results were produced by Edge0 rather than an independent evaluator.

On the M4 Pro test machine, Edge0 reports 14.9-17.7 tokens per second for the 35B model and 23.9-25.3 tokens per second for the 8B model. The smaller release used 1 GB of peak active memory for short contexts. Edge0's own documentation warns that memory rises with context length: a roughly 3,300-token context pushed the 8B model's cache requirements to about 3.3 GB.

The phone still needs its runtime

The distinction between "phone-class memory" and software that developers can run on an iPhone matters. Thursday's package lets developers reproduce the storage-streaming approach on a Mac, inspect the inference code and download the adapted checkpoints. It does not yet give them the iOS application shown in Zeng's video.

Edge0 is also entering a field where storage-backed inference is already established. Apple's "LLM in a Flash" research described loading model parameters from flash memory on demand to run models larger than available DRAM. Open-source projects including Swiftlet have since shipped Swift and Metal implementations that stream sparse model experts from an iPhone's storage.

Edge0's contribution is the combination of storage offload, learned routing prediction and adapters trained around the quantized deployment format. The open release makes that system inspectable. Reproducing the iPhone figure will require the missing mobile backend, along with device-specific measurements for speed, thermals, storage traffic and memory pressure. Those figures will determine whether Edge0's phone demo becomes a usable developer runtime or remains the most effective possible advertisement for its Mac code.