ONNX Runtime cuts single-layer DeepSeek-style CPU decode to 3.45 ms in test
The dedicated operator also halves the tested FP16 cache to 1,152 bytes per token per layer.
By RuntimeWire Staff · Published

The dedicated operator also halves the tested FP16 cache to 1,152 bytes per token per layer.