Supermicro, IBM and NVIDIA put KV cache on shared storage and claim 56x faster first tokens
Supermicro, IBM and NVIDIA say persistent KV cache on Petascale storage delivered up to 56x faster first tokens when long context was reused at 130,000-token prompts in joint testing.
By RuntimeWire Staff · Published · Updated
Primary source: PR Newswire Business Technology
Why it matters
Long-context and agentic workloads can require models to rebuild the same context again and again, using GPU cycles before the first token appears. A persistent, shared KV cache tier gives operators a way to reuse that work across requests and nodes, which can raise GPU utilization and concurrency without buying GPUs in proportion to context growth.
Supermicro founder Charles Liang (@charlesliang) is pushing his company deeper into the inference stack, this time with a storage system built to stop GPUs from redoing work they have already done. On November 5th, 2026, Supermicro announced a KV cache solution with IBM and NVIDIA that combines Supermicro Petascale all-flash storage servers, IBM Storage Scale software, NVIDIA Dynamo inference software and NVIDIA Spectrum-X Ethernet networking.…
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.