The Q4 change was a layout rewrite, not a new decode path.

Caterpillar's four-bit weights were repacked around eight-row tiles, and the scale data now gets unpacked once at load instead of being pulled back through the hot loop on every token. That sounds like a small structural change, but on test hardware it cut bytes read per token by about 13% and improved decode by about 8% while keeping the output byte-identical.

The practical lesson is simple: once decode is memory-bound, the difference between “prepare it once” and “touch it every token” starts to show up as throughput. The math did not change. The answer did not change. The memory traffic did.

That is the part worth watching in your own kernels and data paths. If a format change makes the hot loop carry less baggage, you often get the speedup without any behavioral risk.

Have you seen a layout change beat a kernel rewrite on a workload you care about?

Engineering fact: On Lizard's test hardware, Caterpillar's eight-row Q4 layout reduced bytes read per token by about 13% and improved decode by about 8% with byte-identical output.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#Caterpillar #Q4 #LLMInference #SystemsProgramming #PerformanceEngineering

<!-- lizard-marketing-slot:day-04-pm -->