Hugo Vergnes trained a 3.8B model for a reported $998

Why it matters

Vergnes credits ClimbMix, Muon, FP8, vocabulary padding and rented B200 GPUs for the convergence and throughput behind his reported 3.8B-parameter run. The $998 figure is a reported run cost, not a documented all-in project cost, because the write-up does not itemize development labor, failed experiments, storage, data preparation or other overhead.

Hugo Vergnes trained a 3.848 billion-parameter language model from random weights for a reported $998 in rented GPU time. He says he wrote the project in the evenings.

In a technical write-up, Vergnes reports that the 3.848B model scored 0.384 on CORE after training on 65.3B tokens for 43 hours on eight B200 GPUs at a reported cost of $998. Vergnes's comparison table lists OpenAI's original GPT-2 checkpoint at 0.2565 and cites a 0.310 result for a roughly 1 billion-parameter nanochat run with a similar $1,000 compute bill.

Vergnes reports a $998 cost for the 2,048-token-context run, but the write-up does not itemize whether that figure includes failed experiments, storage, data preparation or evaluation overhead. It also does not establish who funded the compute or account for the economic value of his development time. The $998 should be read as a reported run cost; the project's all-in economic cost is undocumented.

Vergnes developed the training framework in the evenings, debugged it on an RTX 5090 and rented the B200 node only after doing the systems work locally. His public profile lists previous research roles at Whiterabbit.ai and Stanford. Stanford's 2022 commencement program lists him as receiving a master's degree in statistics, while a Whiterabbit.ai patent application names him among the inventors of technology for accelerated radiological screening.

That background shows in the project. Vergnes treated model training as an infrastructure problem before spending heavily on compute, building a configuration-driven framework where the model, dataset, optimizer, schedule and callbacks could be changed through YAML files. His stated goal was to watch language capabilities emerge from random weights and learn the parts of pretraining that packaged frameworks conceal.

Six days spent learning what failed

The strongest part of Vergnes's account is the failed run that preceded the $998 result. He trained an 858 million-parameter Llama-style model on FineWeb-Edu for 16.4 billion tokens, using one A100 for 5.8 days. It scored 60.45% on PIQA, below the roughly 63% result he cited for the 124 million-parameter version of GPT-2. Its generated text was repetitive, and its loss curve had stopped improving well before the run ended.

Vergnes traced the failure to a conservative peak learning rate, a cosine schedule that reduced the rate to zero, the use of AdamW across every parameter class and a dataset that converged too slowly for the budget. The final recipe replaced the cosine schedule with a trapezoidal one, preserving a useful learning rate later into training. It used Muon for matrix parameters and AdamW elsewhere, and replaced FineWeb-Edu with NVIDIA's Nemotron-ClimbMix data.

He also moved the main matrix multiplications to FP8 and padded the tokenizer vocabulary from 50,257 to 50,304 entries, a multiple of 64 that better fits the GPU's tensor-core path. Vergnes attributed a cumulative 33% throughput increase largely to FP8 and vocabulary padding.

Those choices mattered because every inefficiency was multiplied across eight rented accelerators. Vergnes's account of the 1,024-token-context run reports steady-state throughput of roughly 480,000 tokens per second. That run processed 57.3 billion tokens in 35.9 hours and cost $820.

A larger model without a research lab

The final architecture used 28 decoder layers, 24 query heads and eight key-value heads, along with RMSNorm, rotary position embeddings, grouped-query attention, QK normalization and squared ReLU feed-forward layers. Vergnes also allocated 721.2 million parameters to value embeddings placed on alternating layers. Those tables represented about 19% of the model's parameters. Vergnes says they required memory and optimizer state while contributing "essentially no FLOPs."

In an ablation at 12,500 steps, Vergnes reports that the model with value embeddings reached a CORE score of 0.3147 and a loss of 2.1075. The trade-off was memory: the additional parameters required optimizer state and GPU capacity.

The framework borrowed heavily from Andrej Karpathy's nanochat, a minimal training stack covering tokenization, pretraining, fine-tuning, evaluation and inference. Nanochat's current work has pushed the cost of reaching GPT-2-level CORE performance below $100 on an eight-H100 node. Vergnes aimed at a different point on the curve: a considerably larger base model trained for about $1,000.

The benchmark needs context

The 0.384 score needs context. Vergnes says most of the difference between the 1,024-token and 2,048-token runs came from context-sensitive tasks. The result remains a first-party benchmark claim without an independent reproduction.

Vergnes's score reflects the CORE tasks under his reported evaluation setup. It does not establish deployment quality, safety, inference efficiency or how the model would perform after instruction tuning. Vergnes's more durable contribution is the documented training recipe, including the mistakes, ablations and hardware measurements behind the final result.

A $998 final run still required substantial engineering work, local debugging and access to rented top-end hardware. In his account, Vergnes presents the project as evidence that a single engineer can conduct a billion-parameter pretraining experiment at this scale. The benchmark, cost and throughput results remain first-party claims without a cited independent reproduction.