Base Compute gives AI agents the kernel-tuning job for open models

Why it matters

Open-weight releases are useful locally only when runtimes catch up. Base Compute is betting agentic kernel work can compress that lag and make per-chip support economically repeatable.

Lukas Wesemann, Prabod Rathnayaka and Fabian Waschkowski published results on August 20 from a system that assigns AI agents the low-level work of porting open-weight models to new processors and tuning their kernels.

The three Base Compute co-founders call the system the Base Optimization Stack, or B:OS. Their immediate pitch is to model developers, chip vendors and device makers that need a particular model to run efficiently on particular hardware. Each new architecture can otherwise require another round of operator implementation, numerical verification and device-specific optimization by systems engineers.

That repetition is the business problem Base Compute wants to turn into software. Base Compute is trying to preserve what its agents learn from one optimization job and apply those findings to later models and processors. If the approach works across enough combinations, a small inference team could support more releases without treating every port as a fresh consulting project.

Wesemann came to the problem after a physics PhD and nanotechnology postdoctoral research at the University of Melbourne, followed by work on the Australian language models Matilda and Maincoder at Maincode. Rathnayaka, Base Compute's CTO, completed a PhD in artificial intelligence at La Trobe University and worked on Matilda after earlier roles at Replica Studios and John Snow Labs. Waschkowski's mechanical-engineering doctorate used evolutionary machine learning and computational fluid dynamics before he moved into inference research.

Their stated motivation is ownership of the computing stack. Base Compute, an AI inference company operating in Melbourne and Berlin, says its mission is to "run AGI on-device." B:OS is meant to help Base Compute keep local runtimes current as model designs and processor targets change.

The agents introduce a new cost line

Base Compute's experiment also measured what it costs to let different models search for faster implementations. Base Compute gave Claude Fable 5, Kimi K3 and GLM 5.2 the same completed Nemotron port, evaluation setup and eight-hour limit.

In the company's reported results, Fable completed 50 experiments and produced the highest score for $265. Kimi ran 21 experiments, reached 97% of Fable's performance and cost $77. GLM completed 12 experiments and reached 86% of Fable's result while running through BaseRT on an M3 Ultra Mac Studio. Base Compute assigned that local run near-zero marginal cost.

Those figures put an operating decision inside the product. Expensive hosted agents can be reserved for model-hardware combinations where the last increment of throughput matters. Cheaper or locally served models can handle jobs where lower optimization cost outweighs the performance difference. In Base Compute's test, all three agents identified early changes involving state-space-model prefill, decode dispatch and the Mamba convolution path before producing different final results.

The experiment does not establish what customers will pay for automated optimization, and Base Compute has disclosed no revenue, contracts or pricing for B:OS. It does show one possible cost structure for a service that would otherwise depend heavily on scarce kernel engineers.

BaseRT gives the agents somewhere to ship

B:OS produces releases for BaseRT, the native C++ inference runtime Base Compute introduced on July 1. According to Base Compute's BaseRT release article, the runtime targets Apple's Metal API directly and uses handwritten kernels, fused operators and hardware-specific dispatch instead of passing execution through MLX, PyTorch or Core ML.

The public BaseRT repository contains command-line tools, the model format, a public C API and bindings for Python, Node, Rust and Swift under the Apache 2.0 license. BaseRT can also serve an OpenAI-compatible API, allowing local applications to use a familiar request format.

That packaging gives developers a public runtime surface, while B:OS handles some of the model- and hardware-specific engineering needed upstream.

Nemotron supplied the difficult test case

For its first public demonstration, Base Compute chose NVIDIA's Nemotron 3 Nano 30B-A3B, which BaseRT did not previously support. The model has 52 layers and 31.6 billion parameters, with about 3.2 billion active for each token. Its architecture mixes Mamba-2 state-space blocks, conventional attention and a sparse mixture-of-experts feed-forward path.

The port required implementations for selective scanning, causal convolution, normalization, routing and expert computation. After establishing a functioning baseline, the agents searched for faster kernels and execution paths. Base Compute reported that prefill throughput at a 512-token context increased from 85.7 to 927 tokens per second. At a 4K context, prefill rose from 85.4 to 1,093 tokens per second, while decode moved from 85.7 to 111 tokens per second.

Base Compute's harness rejects changes that fail unit tests or push held-out perplexity beyond its specified tolerance. Base Compute said the tuned Nemotron build finished with perplexity 2.0% below the initial port's result. That accuracy gate matters because throughput gains are of little use if an optimization changes model behavior beyond an accepted range.

In the final runtime comparison, BaseRT decoded at 113.8 tokens per second, against 60 for llama.cpp and 79.6 for MLX. Across the measured prefill configurations, Base Compute put BaseRT at 1.39x to 1.76x llama.cpp's throughput and 1.98x to 2.55x MLX's. The resulting decode ratios were 1.90x over llama.cpp and 1.43x over MLX.

Base Compute selected the model, designed the protocol and ran the tests; the results have not been independently replicated. Base Compute says it controlled for hardware, timing and software revisions across the three runtimes. llama.cpp and MLX continue to change, so the comparison records one test setup rather than a durable ranking.

B:OS records the work, not just the winning kernel

Inside B:OS, each attempted modification becomes a git commit tied to a benchmark result. The system starts from public model weights, constructs a Q4 or Q8 bundle, implements missing architecture support and tests optimization candidates before producing a model-and-device-specific BaseRT build.

The agents explore multiple hypotheses rather than following one fixed optimization recipe. Base Compute stores the measured outcome and technical finding from each attempt in memory that can be consulted on later runs. The product value depends on whether those records shorten future work across genuinely different architectures and processors.

Base Compute has begun testing that premise beyond Macs. Its BaseRT changelog lists an NVIDIA GB10 backend alongside Apple silicon. In the B:OS report, Base Compute said knowledge collected during the Apple optimization reduced the end-to-end time for the corresponding Nemotron work on an NVIDIA DGX Spark by 27%.

That 27% figure is another Base Compute measurement, but it addresses a more consequential claim than winning one benchmark sweep. Reusable optimization knowledge could lower the cost of adding each subsequent model and chip. Without that transfer, B:OS remains an automated way to conduct individual tuning projects.

Open weights make a checkpoint available for download; useful local performance still depends on runtime support and low-level processor work. Base Compute's three founders are betting agents can absorb enough of that burden to make hardware-specific releases faster and cheaper to produce. Nemotron provides a Base Compute-run demonstration with a substantial claimed speedup. Independent testing and further ports will determine how much of the result belongs to the stack and how much belongs to this particular model, runtime and machine.