Baseten ships Rust tokenizer it says is 18x faster for Kimi K3

Why it matters

Million-token agent workloads can move latency away from the GPU and into prompt preparation. Baseten is competing by optimizing that overlooked CPU path and releasing the code for engineers to test.

Baseten model performance engineer Michael Feil (@feilsystem) said in a thread on X on July 27th that he had shipped Baseten Tokenizer, a Rust-backed package built to cut the CPU time required to prepare long prompts for Moonshot AI's Kimi K3 model.

The package, named basetenkenizer, was available through PyPI and a Kimi K3 tokenizer repository on Hugging Face when Baseten published its results. PyPI lists Feil as a verified maintainer and shows version 0.2.8 uploaded on July 27th, following two releases on July 26th.

Feil is a machine learning engineer in San Francisco with a computer science master's degree from the Technical University of Munich. According to his personal site, he previously worked on model training at Gradient.ai and created Infinity, an open-source framework for embedding inference. Baseten's author page identifies him as a model performance engineer whose earlier work included embedding throughput, cache-aware routing and Kimi K2.5 serving.

Baseten was founded in 2019 by CEO Tuhin Srivastava, CTO Amir Haghighat, chief scientist Phil Howes and Pankaj Gupta. The founders built Baseten around a problem they had encountered while deploying machine learning systems: training a useful model could be easier than operating it under production traffic. Baseten now sells managed inference, post-training and model optimization, putting Feil's tokenizer work directly inside the San Francisco company's core product thesis.

Tokenization becomes visible at one million tokens

Tokenizers convert text and structural markers into the numerical token IDs a model processes. That step has usually consumed little time compared with loading a prompt onto GPUs and generating an answer. Kimi K3 changes the balance because Moonshot AI's model card specifies a context window of 1,048,576 tokens, a 160,000-token vocabulary and a mixture-of-experts architecture with 2.8 trillion total parameters and 104 billion active parameters.

Agent workloads can repeatedly append documents, tool outputs, code and intermediate state to that context. Prefix caching can reduce the GPU work required to process material the model has already seen, leaving CPU-side tokenization as a larger share of time to first token.

In a technical post published July 27th, Baseten said Basetenkenizer made its complete Kimi K3 serving path more than six times faster than OpenAI's tiktoken on short sequences and as much as 18 times faster on million-token inputs. Those figures are Baseten's benchmarks rather than independent results.

Baseten's implementation replaces a path that looped through prompt segments in Python with one native Rust call. The tokenizer accepts typed (text, allow_special) segments, performs pre-tokenization and byte-pair encoding, and returns a NumPy-compatible token array without first creating a Python list containing as many as one million integers.

The segment distinction is required for prompt integrity. Kimi K3 uses strings such as <|open|> and <|close|> as control markers in structural sections. The same characters inside user text must remain ordinary text. Basetenkenizer keeps that boundary while preserving the token IDs produced by the previous serving path, according to Baseten.

Feil said in his X thread that most competing libraries do not use more than one CPU core for this work. Basetenkenizer uses Rayon thread pools sized at eight or 16 workers and limits concurrency so repeated pre-tokens can produce more cache hits before byte-pair encoding.

Baseten's fastest claim has a narrow scope

Baseten's broadest claim requires qualification. The benchmark used a rendered Kimi K3 conversation containing 54 typed segments and ran on 52 pinned Intel Xeon vCPUs. Baseten generated a new deterministic corpus for every measured round, preventing an entire prompt from being served from a warm tokenizer cache. Tiktoken's public encode call is single-threaded, while Basetenkenizer parallelizes across the available CPU set.

Baseten also reported that gigatoken was faster at 10,000 tokens and that gigatoken remained the better tool for offline file tokenization. At one million tokens, Basetenkenizer led gigatoken by 1.41 times on Baseten's online-serving test. The supported conclusion is narrower than the world's-fastest label in Feil's post: Baseten measured the fastest complete path for Kimi K3 prompts that had already been rendered into typed segments for online inference.

That specialization fits Baseten's position in a crowded inference market. GPU capacity is available from a growing group of platforms, pushing vendors to compete on the software surrounding the accelerator: schedulers, caches, kernels, networking and now tokenization. Day-zero support for a newly released open-weight model also gives Baseten a chance to capture workloads before engineering teams settle on another serving stack.

Baseten has substantial capital for that competition. On June 22nd, Baseten announced a $1.5 billion Series F at a $13 billion valuation, led by Altimeter Capital, Conviction Partners and Spark Capital, with Sands Capital and Wellington Management as co-leads. Baseten said at the time that revenue had grown 20 times and inference volume 40 times over the prior year, figures that Baseten has not supported with underlying financial statements or absolute values.

Basetenkenizer is licensed under MIT and supports Python 3.10 or newer. Its release turns a piece of Baseten's internal Kimi K3 optimization work into a package other inference engineers can install, benchmark and challenge on their own hardware.