llama.cpp adds an x86 VNNI Q2_0 kernel and triples tested CPU throughput

Contributor Saejin Kim-Heinert's group-64 kernel was merged on November 15th, with Bonsai gains from 3.0x to 3.64x and a small perplexity drift attributed to compiler behavior.

By · Published

A fluffy llama sits beside an upright computer processor on a desk.

Contributor Saejin Kim-Heinert's group-64 kernel was merged on November 15th, with Bonsai gains from 3.0x to 3.64x and a small perplexity drift attributed to compiler behavior.