lizard-native: the whole inference stack, running locally

[lizardadmin]

If Lizard is an inference provider, the fair question is which parts of the stack it actually owns. Here is the answer, top to bottom.

The API layer speaks the OpenAI Chat Completions format. This is deliberate: it means the client libraries you already use work by changing a base URL, and it means you are not locked in — anything you build against Lizard runs against a hosted provider too, and vice versa.

Scheduling and batching is where a lot of local runtimes lose time. The naive approach submits work to the GPU many times per token, and each submission costs you. lizard-native compiles the decoder graph into coalesced spans ahead of time and gets that down to roughly one submission per generated token.

The KV cache is resident and its precision is yours to pick — f16, q8_0, q4_0. This is the setting nobody exposes and everybody should: it is a direct trade between context length and memory. A hosted provider makes that choice for you and you never see it.

The kernels are where the model's weights get multiplied. These are quantization-aware — Q4_K, Q5_K, Q6_K — with the elementwise work fused into the matrix multiply so a token costs fewer passes over memory.

The device layer is Direct3D 12, and the important word is resident: the weights are uploaded once and stay in GPU memory. Loading is the expensive part; after the first time, the shaders are cached on disk too, so a load is seconds rather than a minute.

Why this matters for a developer: every one of those layers is a decision that determines your tokens per second, and on a hosted API all of them are made somewhere you cannot see or change. Locally, they are yours — the KV cache precision and the quantization level are literally settings in the app.

The practical ceiling is memory bandwidth, not arithmetic. Generating a token means reading the whole model out of memory, so on most machines the model's size divided by your memory bandwidth is the speed limit. That is also why a smaller quantization is often faster, not just smaller.