Kog says it hit 3,000 tokens/s per request on standard GPUs
Kog opened a live coding playground and argues single-request decode speed, not FLOPS, is the bottleneck that matters for autonomous agents.
By Ryan Merket · Published
Primary source: Kog
Why it matters
If agents are gated by sequential token generation, the product frontier shifts from bigger models to faster loops. Hitting 3,000 tokens/s on commodity GPUs could unlock real-time agent UX without new silicon.
Kog says it has reached 3,000 tokens per second per request on standard datacenter GPUs with a latency-first inference stack, unveiling the results in a blog post and a live demo in the Kog live coding playground.
Kog frames the benchmark around autonomous agents, where single-request decode speed governs loop time for plan-write-test-revise cycles. In their example, an agent generating 50,000 tokens drops from roughly eight minutes at 100 tokens/s to under twenty seconds at 3,000 tokens/s. The public preview focuses on batch size 1 and a 2B-parameter coding model tuned for software tasks, prioritizing speed over scale.
Technically, Kog argues that low-batch autoregressive decoding is memory-bandwidth bound, not FLOPS bound. They emphasize Memory Bandwidth Utilization as the key metric and point to the effective HBM bandwidth available on multi-GPU nodes. On paper, an 8x NVIDIA H200 node at about 30.7 TB/s and an 8x AMD MI300X node at about 33.6 TB/s imply speed-of-light bounds near 7,700 to 8,400 tokens/s for a 2B FP16 model, with similar reasoning extending to MoE setups. Their latency-first design parallelizes inference across the full node to chase that ceiling.
Kog positions this approach as a path to dedicated-inference-card speeds without proprietary silicon lock-in, targeting hardware enterprises, labs, and sovereign-AI buyers already operate. The post also notes next-gen GPUs (Rubin and MI450) are expected to lift memory bandwidth further, raising the ceiling for larger models or fewer GPUs.