Kog says it hit 3,000 tokens/s per request on standard GPUs
Kog opened a live coding playground and argues single-request decode speed, not FLOPS, is the bottleneck that matters for autonomous agents.
By Ryan Merket ยท Published
Why it matters
If agents are gated by sequential token generation, the product frontier shifts from bigger models to faster loops. Hitting 3,000 tokens/s on commodity GPUs could unlock real-time agent UX without new silicon.
Kog says it has reached 3,000 tokens per second per request on standard datacenter GPUs with a latency-first inference stack, unveiling the results in a blog post and a live demo in the Kog live coding playground.
Kog frames the benchmark around autonomous agents, where single-request decode speed governs loop time for plan-write-test-revise cycles. In their example, an agent generating 50,000 tokens drops from roughly eight minutes at 100 tokens/s to under twenty seconds at 3,000 tokens/s. The public preview focuses on batch size 1 and a 2B-parameter coding model tuned for software tasks, prioritizing speed over scale.
Technically, Kog argues that low-batch autoregressive decoding is memory-bandwidth bound, not FLOPS bound. They emphasize Memory Bandwidth Utilization as the key metric and point to the effective HBM bandwidth available on multi-GPU nodes. On paper, an 8x NVIDIA H200 node at about 30.7 TB/s and an 8x AMD MI300X node at about 33.6 TB/s imply speed-of-light bounds near 7,700 to 8,400 tokens/s for a 2B FP16 model, with similar reasoning extending to MoE setups. Their latency-first design parallelizes inference across the full node to chase that ceiling.
Kog positions this approach as a path to dedicated-inference-card speeds without proprietary silicon lock-in, targeting hardware enterprises, labs, and sovereign-AI buyers already operate. The post also notes next-gen GPUs (Rubin and MI450) are expected to lift memory bandwidth further, raising the ceiling for larger models or fewer GPUs.