AMD releases Instella-MoE to showcase end-to-end training on Instinct GPUs

Why it matters

Instella-MoE gives AI teams a reproducible test of AMD's stack for sparse-model training and RL, though its research-only weight license limits commercial use.

AMD researchers led by Jiang Liu and Prakamya Mishra (@PrakamyaMishra) released Instella-MoE on July 24th, giving developers the weights and training artifacts for a mixture-of-experts language model built from scratch on AMD Instinct GPUs. Mishra described the release in a thread on X on Monday as AMD's first fully open MoE model.

The release puts two AMD researchers behind the work in public view. Liu, Instella-MoE's project lead, joined AMD after completing a doctorate in electrical and computer engineering at Johns Hopkins University in 2024. Mishra is an applied research engineer on AMD's Seattle-based GenAI group, where he works on large-scale training techniques. He previously researched natural-language processing at UMass Amherst and worked with Amazon's NLU and spoken-language understanding groups.

Instella-MoE contains 16 billion parameters, while its sparse architecture activates 2.8 billion for each token. Its 27 decoder layers use two shared experts and route each token through six of 64 available experts, a structure intended to preserve model capacity without incurring the compute cost of activating every parameter for every token.

AMD says it trained the model on 7.1 trillion tokens using Instinct MI300X and MI325X accelerators, ROCm and its Primus training framework. The data mixture covered general web text, code, mathematics and science. AMD then ran separate mid-training, long-context, supervised fine-tuning, direct preference optimization and reinforcement-learning stages. Long-context training expanded the supported window from 4,000 to 64,000 tokens.

The scope of the release matters as much as the final checkpoint. AMD published a six-model collection on Hugging Face covering pretraining, mid-training, long-context base, supervised fine-tuning, preference optimization and reinforcement learning. The Instella-MoE repository includes training configurations, data-mixture details, inference tooling and code for reproducing the pipeline.

That package turns Instella-MoE into a reference implementation for training modern sparse models on AMD hardware. Model releases from chipmakers serve a commercial purpose even when the model itself is research-oriented: they give prospective customers working code, performance data and an architecture they can inspect while evaluating an alternative to the dominant Nvidia software stack.

AMD tests the full training pipeline

Instella-MoE also gives AMD a vehicle for two systems techniques developed around the communication costs of mixture-of-experts models. The first, Gated Multi-head Latent Attention, adds a learned gate that controls how much of each attention output moves forward. The second, FarSkip-Collective, overlaps communication and computation across GPUs during expert-parallel training and serving.

AMD reports that FarSkip-Collective increased pretraining speed by 12.7%. AMD also measured reductions of as much as 39.2% in time to first token when serving the model with expert parallelism. Those figures come from AMD's own tests and depend on the specified parallel serving configuration.

The benchmarks are also company-reported. AMD says the base checkpoint averaged 76.7 across its selected standard evaluations, ahead of the fully open OLMo-3-7B and OLMoE-1B-7B models included in its comparison. AMD's final Think checkpoint, refined with reinforcement learning, produced a 73.22 average in the post-training table and an 83.70 score on IFEval. The comparison placed Instella-MoE behind Qwen3.5-4B-Base on the base-model average while AMD's Think checkpoint led the fully open models selected for its post-training table.

The model's openness also carries a licensing boundary. AMD released the training code under the MIT License, while the checkpoints use a ResearchRAIL license limited to academic and research purposes. Developers can inspect the recipes, data mixtures and intermediate weights, but the model license is not an unrestricted commercial open-source grant.

AMD also warns that the checkpoints carry no safety assurances, have not been tested for multilingual performance and are unsuitable for safety-critical, medical or high-accuracy applications. Those constraints make Instella-MoE a research artifact rather than a production model pitched for immediate deployment.

For AMD, the most consequential output is evidence that ROCm and Instinct accelerators can carry an MoE project through pretraining, preference tuning and reinforcement learning. Instella-MoE gives outside teams the artifacts to test that claim on the code and checkpoints instead of relying solely on hardware benchmarks.