Neuralwatt says cache misses can use up to 140% more inference energy
CEO Chad Gibson's analysis of Neuralwatt Cloud traffic links prompt edits to wasted GPU work, with the biggest costs in long agent sessions.
By Ryan Merket · Published
Primary source: Neuralwatt
Why it matters
Neuralwatt's traffic analysis makes the agent harness an efficiency variable alongside models and GPUs. If the findings hold across other deployments, prompt construction gives teams a software-level way to cut repeated inference work, especially in long-running agents.
Neuralwatt CEO Chad Gibson says the software that sends prompts to an AI model can nearly double the energy required to serve a request. In an analysis published October 16th, Neuralwatt compared prompt-cache reuse across agent clients and found that small differences in how they assemble prompts can translate into substantial extra prefill work.…