Baseten's inference essay separates real efficiency gains from cost shifts

Why it matters

Baseten's $13 billion valuation depends on inference optimization remaining a defensible engineering business rather than collapsing into commodity model hosting. Its framework gives developers a practical way to tell real efficiency gains from latency, quality or cost tradeoffs hidden behind a benchmark.

On September 1st, Baseten - founded by Tuhin Srivastava (@tuhinone), Amir Haghighat, Phil Howes and Pankaj Gupta - published a technical framework for separating genuine improvements in large language model inference from the tradeoffs that merely shift costs elsewhere.

The distinction sits at the center of the founders' bet on Baseten. Srivastava and Howes previously worked together at Gumroad and co-founded people-analytics startup Shape, which Reflektive acquired in 2018. Haghighat was an early Gumroad engineer before leading machine-learning engineering at Clover Health, while Gupta brought infrastructure experience from Uber and Twitter. They started Baseten in 2019 after repeatedly watching capable machine-learning models become slow, fragile production systems.

Seven years later, Baseten is valued at $13 billion and has considerably more money riding on the proposition that inference deserves its own infrastructure layer. Baseten raised a $1.5 billion Series F on June 22nd, led by Altimeter Capital, Conviction Partners and Spark Capital, with Sands Capital and Wellington Management as co-leads. Baseten said the financing would fund compute, software and hiring.

The new essay, written by lead developer advocate Philip Kiely, explains what that software work is supposed to accomplish. Kiely, whom Baseten identifies as its 10th employee, divides inference engineering into two buckets: configuration decisions that move a deployment along an existing performance frontier, and technical advances that push the frontier outward.

That framing sounds academic. In production, it is a method for deciding whether an engineering change created value or moved the bill.

Some optimizations send the cost somewhere else

For most LLM deployments, Baseten defines the central frontier as the tradeoff between latency and throughput. Serving each user faster usually leaves less room to process other requests on the same GPU. Packing more requests together can lower the cost per token while making each user wait longer.

Batch sizing is the clearest example. Small batches prioritize per-request speed and leave expensive GPU capacity less fully used. Larger batches improve total token throughput and reduce unit costs, while individual requests take longer. Neither setting is universally better. A consumer voice product and an overnight document-processing pipeline have different economic targets.

Parallelism creates similar choices. Large models must be spread across GPUs, and the method used to divide the work affects both communication overhead and utilization. Baseten says tensor parallelism is generally useful when latency is the priority, while wide expert parallelism and attention data parallelism can favor throughput.

Quantization complicates the accounting. Running model weights, activations or cache values at lower precision can improve speed and throughput together. It also introduces a separate frontier between serving efficiency and model quality. The acceptable point depends on the application and requires workload-specific evaluation rather than a generic claim that a lower-precision model is cheaper.

The essay's useful discipline is its refusal to treat every optimization as a free gain. A deployment that doubles throughput by accepting slower responses has found a different operating point. That can produce a better business outcome, especially for batch workloads, but it has not made the underlying system twice as efficient for every customer.

The gains Baseten wants to sell

Baseten assigns greater value to work that improves the whole curve. Faster kernels and runtimes can reduce the compute needed to generate each token. Better hardware can increase the available performance budget. Speculative decoding can generate likely tokens with a smaller process and verify them with the primary model, skipping some expensive forward passes when predictions are accepted.

Prefill-decode disaggregation separates the phase that processes a prompt from the phase that generates output tokens. Each phase can then run on workers configured for its own compute and memory demands. Baseten argues that this is most useful for increasing throughput while maintaining or slightly improving latency in high-volume deployments.

Those techniques can compound. A hardware improvement and a software improvement may multiply rather than merely add, although the resulting gain remains dependent on model architecture, sequence length, cache-hit rate and traffic patterns. The essay provides no independently audited benchmark for Baseten's entire inference stack, so it should be read as an engineering framework rather than proof of a universal cost advantage.

That limitation matters because Baseten is using the essay to support a product pitch. It assumes workloads running models such as GLM-5.3 or Kimi K3 for agentic coding, with KV-cache reuse and cache-aware routing already enabled. Baseten began offering GLM-5.3 through its Model APIs and training tools on August 28th.

Baseten lists GLM-5.3 at $1.40 per million input tokens, $0.14 per million cached-input tokens and $4.40 per million output tokens. Baseten describes the Z.AI model as a 753-billion-parameter mixture-of-experts system with 40 billion active parameters. The endpoint includes low, high and maximum reasoning settings, giving developers another explicit tradeoff between response speed and test-time computation.

Baseten also says GLM-5.3 improved from 4.6% to 28.3% over GLM-5.2 on Terminal-Bench 3.0. That figure comes from Baseten's product page rather than an independent evaluation, and it measures model performance rather than the efficiency of Baseten's serving infrastructure.

Inference has become a heavily financed category

Baseten's framework arrives amid an expensive contest to own the production layer beneath AI applications. Modal raised $355 million at a $4.65 billion valuation in May and positions itself as a broader AI cloud spanning inference, batch workloads, reinforcement learning and isolated environments for software agents. Inferact, founded by creators and maintainers of vLLM, is building around the argument that inference optimization should continue flowing into an open-source engine.

Baseten's position is narrower and operationally demanding: optimize models for specific traffic, then manage routing, autoscaling and GPU capacity across cloud, single-tenant and self-hosted environments. Baseten's documentation describes active-active deployments across clusters and clouds, with infrastructure that provisions accelerators and monitors their health across providers and regions.

The market is rewarding that pitch. Baseten said in its Series F announcement that revenue increased 20-fold and inference volume increased 40-fold in the year preceding the financing. Those are Baseten-supplied growth rates, without current revenue or customer totals attached. Baseten names Cursor, Notion, Lovable, Harvey, HubSpot, OpenEvidence, Abridge and Decagon among its customers.

The founders' original problem has become larger as models have become more capable. Longer prompts, reasoning tokens, agent loops and mixture-of-experts architectures create additional places for GPU time to disappear. Model access is increasingly interchangeable for open-weight releases. The engineering required to serve those models at a predictable speed and cost remains highly specific to the workload.

Baseten's essay gives buyers a vocabulary for interrogating the resulting claims. A lower price may come from slower responses, lower precision or better cache reuse. A genuine shift in the frontier requires the same deployment to produce more useful work from the same hardware budget. Baseten has raised $1.5 billion to prove its engineers can keep finding those shifts before customers decide an endpoint alone is good enough.