Inferact's TPU megakernel hits 709 tokens per second on Kimi K3
Inferact reports 709 tokens per second on 16 TPU v7 chips, versus 452 on 16 GB200 GPUs, in a low-concurrency speculative-decoding test. Co-founder Woosuk Kwon, a vLLM co-creator, helped build the open-source kernel.
By RuntimeWire Staff · Published
Primary source: Inferact
Why it matters
Inferact is making its hardware-specific inference work public while extending vLLM toward Google TPUs. The 709-token result is a company benchmark under limited conditions; production concurrency, cost per token and independent reproduction will determine its practical weight.
Inferact says its new TPU megakernel served Kimi K3 at 709 tokens per second, versus 452 tokens per second on a 16-GPU GB200 baseline, in a low-concurrency test using speculative decoding. The result comes from an open-source project led in part by Woosuk Kwon (@woosuk_k), Inferact's co-founder and CTO and a co-creator of vLLM.
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.