NVIDIA says new optimizations make local agents up to 1.9x faster
The llama.cpp and vLLM updates target RTX 5090, RTX PRO 6000 and dual DGX Spark systems, with support through LM Studio and Ollama.
By Ryan Merket · Published
Primary source: NVIDIA RTX Spark on X
Why it matters
Local agents make repeated inference calls, so backend gains can reduce latency and cloud usage without requiring developers to replace their existing NVIDIA hardware.
NVIDIA said Friday in a post on X that new llama.cpp and vLLM optimizations can raise local large-language-model throughput by as much as 1.9x across its Blackwell consumer, workstation and desktop AI systems.
The largest gain in NVIDIA's benchmark came from llama.cpp running Qwen3.6-35B on a GeForce RTX 5090, where throughput reached 1.9x the baseline NVIDIA used at Computex. Qwen3.6-27B reached 1.5x the same baseline on the card.
The gains were smaller on NVIDIA's professional and dedicated AI hardware. Using vLLM, Qwen3.6-27B and Qwen3.6-35B both posted 1.2x performance on the RTX PRO 6000 Blackwell Workstation Edition. A cluster of two DGX Spark systems reached 1.4x on Qwen3.6-27B and 1.2x on DeepSeek v4 Flash.
NVIDIA's chart compares its Computex results with software dated September 1st. It identifies the workload as SpeedBench-Coding 8K throughput with a batch size of one, making the figures a measure of relative improvement under NVIDIA's selected test rather than a general performance ranking. NVIDIA did not include absolute token-per-second results or energy consumption in the graphic. The headline 1.9x figure also applies to one model, framework and GPU combination; the remaining tests ranged from 1.2x to 1.5x.
The speedup is coming from the inference stack
In an accompanying IFA 2026 announcement, NVIDIA attributed the llama.cpp gains to optimized kernels, improved speculative decoding and faster prompt prefill. Speculative decoding allows a smaller draft component to propose multiple tokens that the main model can verify together, reducing the number of sequential generation steps when enough proposed tokens are accepted.
The vLLM improvements use new XQA attention kernels in FlashInfer alongside backend changes, according to NVIDIA. FlashInfer's attention documentation includes XQA kernels for decoding against a key-value cache, while vLLM is designed around continuous batching and memory-efficient model serving.
NVIDIA said the updates are available through the llama.cpp and vLLM backends, as well as through LM Studio and Ollama. That distribution matters because developers running local models often encounter these optimizations through packaged applications rather than compiling inference engines and CUDA kernels themselves.
The software work also gives NVIDIA another way to raise the useful output of hardware already in users' machines. The GeForce RTX 5090 is the consumer configuration in the benchmark, while the RTX PRO 6000 carries 96GB of GDDR7 memory for models and workloads that exceed consumer-card capacity. Each DGX Spark has 128GB of unified memory, and NVIDIA supports linking multiple systems for larger deployments.
NVIDIA is widening the local-agent funnel
The performance release sits inside a broader NVIDIA push to make local agents easier to install and run. NVIDIA said Hermes Agent, OpenClaw and Perplexity Portable Computer are adding simplified setup for local models on supported RTX and DGX hardware. The same IFA announcement introduced NVIDIA PAIR, open-source software that routes inference jobs among compatible computers on a local network.
Faster inference strengthens that pitch. Agentic workloads can repeatedly call a model while reading files, using tools and revising intermediate results, so modest delays accumulate across a task. Higher throughput can shorten those loops and increase the number of concurrent jobs a workstation or small cluster can handle.
NVIDIA is also tying the software updates to a broader hardware lineup. RTX Spark Windows PCs from Lenovo, Acer and other manufacturers are scheduled to ship in October 2026. NVIDIA describes those systems as combining a Blackwell GPU, a 20-core Grace CPU and up to 128GB of unified memory in laptops and compact desktops.
For developers, the immediate deliverable is narrower and testable: updated inference backends that NVIDIA says produce between 20% and 90% more throughput in its selected configurations. Reproducing those gains across different model sizes, quantization formats, context lengths and concurrency levels will determine how much of the benchmark survives contact with real local-agent workloads.