NVIDIA's TensorRT-LLM more than doubles DeepSeek cache capacity in one test
NVIDIA's batch-based allocator gave MLA layers 38.9 GiB of a 43 GiB DeepSeek-R1 cache reservation, up from 16.1 GiB in one reported layout.
By RuntimeWire Staff · Published

NVIDIA's batch-based allocator gave MLA layers 38.9 GiB of a 43 GiB DeepSeek-R1 cache reservation, up from 16.1 GiB in one reported layout.