ONNX Runtime cuts single-layer DeepSeek-style CPU decode to 3.45 ms in test

The dedicated operator also halves the tested FP16 cache to 1,152 bytes per token per layer.

By · Published

A green circuit board with a large exposed processor and smaller chips sits inside a metal chamber marked “Rw.”

The dedicated operator also halves the tested FP16 cache to 1,152 bytes per token per layer.