Inferact's TPU megakernel hits 709 tokens per second on Kimi K3

Why it matters

Inferact is making its hardware-specific inference work public while extending vLLM toward Google TPUs. The 709-token result is a company benchmark under limited conditions; production concurrency, cost per token and independent reproduction will determine its practical weight.

Inferact says its new TPU megakernel served Kimi K3 at 709 tokens per second, versus 452 tokens per second on a 16-GPU GB200 baseline, in a low-concurrency test using speculative decoding. The result comes from an open-source project led in part by Woosuk Kwon (@woosuk_k), Inferact's co-founder and CTO and a co-creator of vLLM.

Inferact on X

Kwon and Inferact engineers George Novack, Xuting Liu and Jeff Ma published the benchmark and released the TPU megakernel code on September 23rd. The work puts a specific engineering bet in public: inference speed can improve when software is written to control how model weights move through a chip's memory, rather than leaving the full schedule to general-purpose compiler optimization.

That bet fits Kwon's path into Inferact. He completed a computer science PhD at UC Berkeley, where he was advised by Ion Stoica, and previously worked as a research scientist at Google DeepMind and a member of technical staff at Thinking Machines Lab. At Berkeley, he co-created and co-led vLLM, the open-source inference engine Inferact was formed to develop. Inferact says it intends to contribute its optimizations back to vLLM rather than keeping them exclusively inside a proprietary serving stack.

What the benchmark measures

Inferact's comparison uses 16 TPU v7 Ironwood chips against 16 NVIDIA GB200 GPUs, running Kimi K3 with DSpark speculative decoding. In that setup, the TPU implementation reached 709 output tokens per second, while the GB200 baseline reached 452. The speculative-decoding chart uses an acceptance length of six. Inferact also reports 1,515 tokens per second for Qwen 3.8 27B on TPU v7, against 695 on GB200, under speculative decoding.

Those are Inferact's benchmark results, not an independent reproduction. The figures compare a specific model, hardware configuration and decode workload; they do not establish that TPU v7 is generally faster than GB200. Inferact describes the Kimi test as low-concurrency, and its post says the GB200 comparison uses vLLM's published Kimi K3 recipe. The post does not report cost per token or power consumption, and it does not show the result at production-scale concurrency.

The comparison still points to a concrete systems problem. During decode, inference repeatedly reads model weights from high-bandwidth memory. Inferact's implementation combines much of a model's decode work into one Pallas program and explicitly schedules transfers into TPU vector memory, or VMEM. Inferact says this lets the kernel begin fetching a later layer's weights while the current layer is computing or communicating.

A megakernel bundles operations that would otherwise run as separate kernels. On TPU v7, Inferact's design uses a large software-managed pool of on-chip memory to stage weights ahead of their use. The Kimi implementation combines 92 mixture-of-experts layers in one Pallas call and uses asynchronous copies to overlap memory transfers with computation. The performance claim is therefore as much about the hand-written schedule as about the chip: Inferact is showing what a carefully tuned program can extract from TPU memory and execution behavior.

A focused release, with commercial stakes

The public repository includes Kimi and Qwen implementations, scripts, tests and an OpenAI-compatible server. Inferact says its Kimi kernel can compile from scratch in less than 90 seconds, compared with more than 30 minutes for compiling a large model made up of hundreds or thousands of XLA operations. Faster compilation could shorten the loop for engineers tuning kernels to a particular model and hardware setup; that is a development claim, separate from the throughput benchmark.

Inferact says its non-speculative Kimi K3 and Qwen 3.8 27B megakernels delivered roughly 1.4 to 2 times the GB200 baseline's decode throughput at batch sizes one through eight. Inferact also acknowledges in its post that higher concurrency can move the bottleneck toward vector work and cross-device communication, and that the current implementation is tailored to one TPU chip topology. Those caveats define the next engineering work: scaling the method beyond a small-batch demonstration and adapting it to other TPU arrangements.

The release follows Inferact's September 14th announcement of an engineering partnership with Google focused on making TPU a first-class target for vLLM. It also arrives after Inferact's August announcement that Kimi K3 had passed its vendor verification. Together, those steps give the kernel release a commercial context: Inferact is trying to make its open-source inference expertise useful across hardware vendors, while turning that work into performance its customers can use on their own infrastructure.

Inferact launched in January with a $150 million seed round at an $800 million valuation, according to Inferact. Andreessen Horowitz and Lightspeed led the round, with Sequoia Capital and other investors participating. That financing gives Inferact room to work on infrastructure whose value depends on repeated, model-specific optimization, not a single benchmark. The benchmark supplies evidence for the technical approach; it does not disclose customer economics or show whether the throughput advantage translates into a lower serving bill.

For Kwon and the other vLLM founders, publishing the kernel makes the engineering argument inspectable. The repository offers code and reproduction scripts, while the headline number remains tied to Inferact's own setup and test conditions. Whether that approach holds across workloads and at higher concurrency will determine how far this first result travels beyond a carefully tuned decode test.