Periodic Labs details the 1,300-GPU stack behind Neon

Why it matters

Periodic Labs is treating physical experiments as an RL environment, forcing its founders to build model training, tool execution and laboratory operations as one system.

Liam Fedus (@LiamFedus) and Ekin Dogus Cubuk (@ekindogus) have detailed the reinforcement-learning infrastructure behind Neon, the 1-trillion-parameter model analyzing experiments inside Periodic Labs' high-throughput laboratories.

Aligned News thread

Periodic Labs said in a September 15th engineering post that Neon's final training run used a peak of 1,300 Nvidia H200 GPUs across midtraining and reinforcement learning. Periodic Labs reported more than 95% cluster utilization, 4.1x the training throughput of its Megatron baseline and a 2.5x improvement in inference speed for its workload. An Aligned News thread highlighted Delta Router Replay, one of the narrower engineering fixes inside that larger system.

Those are Periodic Labs' internal measurements. The training comparison uses a Megatron v0.18 recipe adapted for H200s and Periodic Labs' workloads, followed by custom recipe tuning, sequence packing and scheduling. The inference result measures decoding on a 1-trillion-parameter model at the same batch size, improving from 10 to 25 tokens per second per request.

The engineering disclosure shows what Fedus and Cubuk's scientific thesis requires in practice. Periodic Labs wants AI models to reason over experiments for hours, call scientific tools and learn from results produced in physical laboratories. That workload behaves differently from the short prompts and bounded tasks behind much commercial model serving.

The router was resending the whole conversation

Neon uses a mixture-of-experts architecture, which sends each token to a selected subset of specialized neural-network components. Reinforcement learning introduces a synchronization problem: routing decisions made while a model generates a rollout can diverge from the decisions used later during optimization. Router replay preserves the original choices so the training step can reproduce what happened during inference.

Periodic Labs found that the original SGLang implementation returned routing information for the entire conversation after every turn, including tokens already stored in the prefix cache. The payload grew with the conversation even though the rollout client needed only the routing decisions for newly generated tokens.

Delta Router Replay caches the earlier decisions on the client and asks the server only for the new suffix. Periodic Labs contributed the change through SGLang pull request 24851, which added a routed_experts_start_len parameter for specifying where the returned routing data should begin. The pull request was merged on May 10th, several months before Periodic Labs published the broader infrastructure account.

The pull request's benchmark used Kimi-K2-Instruct on eight H200 GPUs, with a 95% prefix-cache hit rate and 100 generated tokens. At a 32,768-token input, returning the full routing history took 121.23 milliseconds. Returning only the new slice took 1.90 milliseconds, a 63.77x speedup for that operation under the stated conditions.

That benchmark measures routing-data collection and transfer, rather than Neon's full end-to-end performance. Its value is in removing a delay that spread beyond one request. In SGLang's data-parallel attention setup, ranks synchronize during decoding, so a single rank stalled while gathering a large routing payload can slow the entire group.

Scientific rollouts break the usual training loop

Periodic Labs said its scientific reinforcement-learning rollouts can reason and execute tools for more than an hour, while a training step takes minutes. Waiting for every rollout before updating the model would leave expensive training hardware idle. Updating too aggressively creates another problem: later portions of a rollout may be generated by different weight versions than the one that started it.

Periodic Labs runs training and inference asynchronously on separate GPU allocations and uses SGLang's in-place mode to preserve the key-value cache during weight synchronization. Prefill-decode disaggregation reduced rollout duration by 2x in Periodic Labs' tests. Together, prefill-decode disaggregation and Delta Router Replay made decoding 2.5x faster for Periodic Labs' use case.

Checkpoint handling created a second bottleneck. Converting a trillion-parameter Megatron checkpoint into Hugging Face format initially took 30 minutes. Periodic Labs distributed the conversion across Ray workers and avoided reconstructing full mixture-of-experts tensors, reducing the process to one minute. Periodic Labs contributed that work to the Miles project.

The physical-science tools required another custom layer. Periodic Labs' first reinforcement-learning loop ran training, inference and model-written code without isolation. According to the engineering post, one generated program requested 80 GB of memory and crashed the job.

Periodic Labs responded by building pbox, a sandbox system that runs on spare CPUs inside GPU nodes already assigned through Slurm. In a 100-sandbox test, Periodic Labs said pbox completed 1 MiB upload-and-download trips with 4.5x lower median latency and delivered 3.3x the aggregate throughput of an unnamed hosted provider. The unnamed comparison and internal test setup limit how broadly those figures can be applied.

The founders are joining the model and the laboratory

Fedus previously led post-training research at OpenAI and worked on ChatGPT-era systems after studying physics at MIT and the University of California, San Diego, and completing a computer science doctorate in Montreal. Cubuk earned a doctorate in applied physics at Harvard, completed postdoctoral research at Stanford and later led materials and chemistry research at Google DeepMind. He was a senior author on GNoME, DeepMind's system for predicting stable crystal structures.

Cubuk told Physics Today in March that experiments were essential because theory and simulations could not carry solid-state physics alone. He also said building a physical laboratory inside a large software company had proved difficult. Periodic Labs gives the founders control of both sides: the models proposing and interpreting work, and the laboratories generating evidence.

That combination attracted a $300 million seed round in September 2025, led by Andreessen Horowitz. Backers included Felicis, DST Global, NVentures, Accel, Jeff Bezos, Elad Gil, Eric Schmidt and Jeff Dean. Felicis partner Peter Deng wrote that he offered the first check before Periodic Labs had a name or had been incorporated.

The new infrastructure account explains where some of that capital is going: a large GPU fleet, customized open-source systems and autonomous laboratories designed as one reinforcement-learning environment. Periodic Labs' reported throughput gains could let its researchers run more experiments within the same compute budget. The scientific test remains the output of that loop. The September post documents how Periodic Labs trains and serves Neon; it does not report a newly validated material or superconductivity result.