Lizard-LLM Community

Inference Engineering · Day 11 · Evening [image: Screenshot%202026-07-22%20221126.png] I think one thing that's easy to misunderstand in LLM benchmarks is tokens/sec. Our latest benchmark showed 40.7 tok/s for Lizard versus 21.8 tok/s for llama.cpp, using the exact same Llama 3.2 3B Q4_K_M GGUF model. The important part is that this is aggregate throughput with 16 concurrent requests (B=16) over multiple real HTTP runs. It's not the speed of a single chat response. Per-user latency and aggregate throughput answer different questions, and I think we should be much clearer about which one we're talking about when comparing inference engines. Whenever I see a tokens/sec number now, my first question is: Is that per response, or total throughput? Engineering fact: On one verified Llama 3.2 3B Q4_K_M run series, Lizard measured a 40.736 tok/s median at B=16 versus 21.755 tok/s for llama.cpp using the same physical GGUF; the result is aggregate HTTP throughput on that test system. Read the relevant Lizard page #LizardLLM #LLMInference #Benchmarking #LocalAI #PerformanceEngineering <!-- lizard-marketing-slot:day-11-pm -->

Inference Engineering · Day 11 · Morning [image: Screenshot%202026-07-22%20221126.png] A fast worker loop is not automatically a fast product. If the benchmark only times an internal execution loop, it can hide the parts users feel immediately: request parsing, scheduling, serialization, and the wait for the full response. Lizard's native readiness benchmark avoids that trap by sending concurrent requests through the OpenAI-compatible HTTP endpoint and measuring full-response throughput on the same path a client actually uses. That matters when you are comparing runtimes, not just kernels. A tight inner loop can look excellent while the request path still pays for queueing, transport, and response assembly. Once you measure the full HTTP path, the numbers are harder to hand-wave away, but they are also more useful for capacity planning and for spotting where latency is really coming from. Internal measurements still have a place. They help isolate decode cost, scheduler overhead, and serialization work. But the user-facing claim should come from the user-facing path. Otherwise you end up optimizing a component and calling it a product result. What is the first thing you add when you want a benchmark to reflect real user wait time instead of a convenient inner loop? Engineering fact: Lizard's native readiness benchmark sends concurrent requests through the OpenAI-compatible HTTP endpoint and measures full-response throughput instead of quoting only an internal worker loop. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #inference #benchmarking #httppath #latency #localai <!-- lizard-marketing-slot:day-11-am -->

lizard-native needs a DirectX 12 capable adapter. In practice that's essentially anything from the last several years, including integrated graphics. Confirmed working so far: Intel Iris Xe (built into many laptops) — works well, memory footprint matters most here NVIDIA and AMD discrete cards — including the case where the model is larger than the card If you'd like to add yours, this format is genuinely useful to others: GPU: e.g. RTX 4060 8GB / Iris Xe Model + size: e.g. llama-3.2-3b Q4_K_M Engine chosen: lizard-native or caterpillar Tokens/sec: from the readiness check You can read tokens/sec straight off the readiness check in setup, no benchmark credits needed.

Inference Engineering · Day 10 · Evening [image: Screenshot%202026-07-22%20222551.png] Ten days of inference engineering ends with the most important benchmark input: your question. Our published results belong to the hardware, model, quantization, prompt and token budget shown. They identify real bottlenecks; they do not create a universal provider ranking. What should we test next? Please include your CPU/GPU, available memory, model, quantization and the decision you need the benchmark to answer. Engineering fact: Published Lizard results are model-, quantization-, prompt-, hardware-, and run-specific; the next useful test should be selected with that context. Read the relevant Lizard page #Benchmarking #LocalAI #InferenceEngineering #LizardNative #CaterpillarEngine <!-- lizard-marketing-slot:day-10-pm -->

Inference Engineering · Day 10 · Morning [image: Screenshot%202026-07-22%20213650.png] If you are trying to keep a local runtime easy to adopt, the API shape matters as much as the model choice. After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint. That means an existing app can usually keep its client logic, swap the base URL, and keep moving. You still get local prompts and outputs, which is the part that usually decides whether a team treats a runtime as a tool or as a project. That compatibility also changes how you test integrations. You can validate request format, retry behavior, and response handling against a local endpoint before you ever point anything at a remote provider. For Windows teams especially, that is a practical way to reduce friction without giving up the familiar /v1/chat/completions style contract. The lesson is simple: local-first works better when it changes the data path, not the application’s expectations. If the endpoint stays familiar, the migration is mostly about deployment and model choice instead of rewriting every client. When you’ve brought a local model endpoint into an existing app, what usually breaks first: auth assumptions, streaming, or tool-calling shape? Engineering fact: After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint while prompts and outputs stay on the machine. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #openaicompatible #windows #llmops #apidesign <!-- lizard-marketing-slot:day-10-am -->

Inference Engineering · Day 9 · Evening [image: Screenshot%202026-07-22%20221220.png] A benchmark dashboard can mislead without containing a false number. Treat an excluded provider as zero. Mix warm-up into decode. Turn one metric winner into a universal recommendation. Each shortcut changes the story. Lizard keeps provider identity, wall time, end-to-end throughput, native decode, load, warm-up and memory separate. Missing providers stay unmeasured. The goal is not to remove judgment. It is to keep the evidence needed for judgment. Engineering fact: Lizard's dashboard keeps excluded providers unmeasured and separates wall time, end-to-end throughput, decode, warm-up, and memory. Read the relevant Lizard page #Benchmarking #DataIntegrity #LLMInference #Reproducibility #LocalAI <!-- lizard-marketing-slot:day-09-pm -->

Inference Engineering · Day 9 · Morning [image: Screenshot%202026-07-22%20213915.png] A signed GPU result must prove that the GPU path actually ran. When a remote benchmark model is larger than a discrete card's memory, a quiet CPU fallback would produce a number—but not the number requested. lizard-native can instead stream weights through the Direct3D 12 decode lane. It is slower than full residency, and the result says so, but the measurement remains a genuine DirectX path with adapter and backend evidence. A slower honest result is more useful than a faster mislabeled one. Engineering fact: For remote benchmarks on a discrete GPU smaller than the model, lizard-native can stream weights through the D3D12 lane rather than silently reporting a CPU fallback as GPU. Read the relevant Lizard page #D3D12 #GPUInference #Benchmarking #LizardNative #LocalAI <!-- lizard-marketing-slot:day-09-am -->

Inference Engineering · Day 8 · Evening [image: Screenshot%202026-07-22%20214040.png] A process ID is not enough to prove process identity. Windows can recycle a PID after a model worker exits. If cleanup logic trusts the number alone, it may block on an unrelated process—or terminate the wrong one. Lizard records PID plus start time and verifies the descendant tree before declaring model memory released. A crashed worker becomes recoverable without making cleanup reckless. Inference reliability includes the process lifecycle around the kernel. Engineering fact: Lizard tracks process identity using PID plus start time so a recycled Windows PID is not mistaken for the old model process. Read the relevant Lizard page #WindowsEngineering #ReliabilityEngineering #ProcessManagement #LocalAI #LizardLLM <!-- lizard-marketing-slot:day-08-pm -->

Inference Engineering · Day 8 · Morning [image: Screenshot%202026-07-22%20213816.png] A missing baseline should produce a failed preflight—not a blank treated as zero. In this run, the requested Ollama model was not installed. Lizard recorded the exact command, failure reason and exit status, then refused to manufacture an Ollama measurement. The later Native, Caterpillar and llama.cpp run is valid for those providers. Ollama remains unmeasured. Reliable benchmarking begins by preserving failure evidence. Engineering fact: When the requested Ollama model is absent, Lizard records the failed preflight and produces no synthetic Ollama measurement. Read the relevant Lizard page #Benchmarking #Reproducibility #LocalAI #LLMInference #LizardLLM <!-- lizard-marketing-slot:day-08-am -->

Inference Engineering · Day 7 · Evening [image: Screenshot%202026-07-22%20213232.png] We implemented an AVX-512 VNNI decode kernel and chose not to enable it by default. The instruction is promising on supported Intel and AMD CPUs. But on the hardware available for validation, the measured result remained inside ordinary run-to-run noise. Caterpillar ships the proven path as default and keeps VNNI behind an explicit experiment flag. Engineering credibility sometimes means declining to claim a speedup. Engineering fact: Caterpillar's AVX-512 VNNI kernel remains opt-in because it measured within normal run-to-run noise on available test hardware. Read the relevant Lizard page #AVX512 #CPUOptimization #Benchmarking #CaterpillarEngine #LizardLLM <!-- lizard-marketing-slot:day-07-pm -->

Inference Engineering · Day 7 · Morning [image: lizard-native-stack.png] KV-cache is one of those pieces that only looks invisible until a long chat starts crowding out memory. Each generated token adds keys and values that have to stay around for later attention, so cache precision is not a trivia setting. It is part of the capacity plan. lizard-native makes that choice explicit with f16, q8_0, and q4_0 KV-cache precision. That matters because the trade is real: preserve more fidelity, or fit more context inside the same machine budget. When the decision is hidden, people tend to blame the model when the actual limit is memory. The useful part is not that one setting is always best. It is that the runtime surfaces the cost where the user can reason about it. On a laptop or a small workstation, that changes the conversation from “why did this stop?” to “what do I want to spend memory on?” In practice, I would rather see a clear cache trade than a vague “supported” label. It gives operators something they can tune against the session length they actually need, instead of discovering the limit after the context has already grown. When you are sizing local inference for real users, do you bias toward higher cache precision or more room for conversation length? Engineering fact: lizard-native exposes f16, q8_0, and q4_0 KV-cache precision as an explicit context-length versus memory trade. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #KVCache #ContextWindow #LocalInference #SystemsEngineering #WindowsAI <!-- lizard-marketing-slot:day-07-am -->

Inference Engineering · Day 6 · Evening [image: lizard-native-stack.png] An integrated GPU is not a small discrete GPU. On unified-memory hardware, CPU and GPU already access the same physical memory. Treating that system like a discrete card can add unnecessary staging and copies. lizard-native includes a UMA zero-copy path for supported integrated graphics. The design goal is simple: shared weights should stay shared, while the Direct3D 12 execution path remains explicit. Hardware-aware inference starts by respecting the memory architecture you actually have. Engineering fact: lizard-native includes a zero-copy path for unified-memory integrated GPUs so shared memory is treated as shared rather than copied as if discrete. Read the relevant Lizard page #LizardNative #IntegratedGPU #D3D12 #ZeroCopy #LocalAI <!-- lizard-marketing-slot:day-06-pm -->

Inference Engineering · Day 6 · Morning [image: Screenshot%202026-07-22%20222539.png] A decode loop can look GPU-bound and still waste time on the host. In the lizard-native Gemma path, roughly 400 weight descriptors were being rebuilt for every generated token. That meant the runtime kept reconstructing the same per-token metadata before the next step could move forward. The GPU had work to do, but so did the scaffolding around it, and that scaffolding sat on the critical path. The fix was to stop rebuilding those descriptors and keep them around for reuse. On the Gemma test runs, that improved decode by 13–26% and brought first-token response back by about 20%. The useful lesson is that decode tuning is rarely only about the kernel. If the loop repeats cleanly, inspect the state that gets recreated every pass: descriptor setup, graph metadata, plan assembly, session bookkeeping, cache wiring. Those are easy to ignore because they do not look like compute, but they can still decide how fast the next token appears. I have learned to ask a boring question before touching the math: what is being rebuilt on every token that could have been carried forward from the previous one? In practice, persistent per-token state often buys a cleaner win than shaving a few cycles off the math itself. Less allocation, less synchronization pressure, fewer host-side stalls, and a shorter path back into the decode loop. When the hot path is stable, that kind of cleanup tends to show up immediately in the first-token feel as well as steady-state throughput. Engineering fact: The lizard-native Gemma path stopped rebuilding roughly 400 weight descriptors per token; test runs improved decode 13–26% and first-token response about 20%. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #inferencetuning #decodeperformance #windowsai #d3d12 <!-- lizard-marketing-slot:day-06-am -->

You don't need a card big enough to hold the whole model. When a model doesn't fit, Lizard has two honest options and it will tell you which one it took: Stream the weights through the GPU. The card still does the work, but it reads parts of the model as it goes. Slower than fully resident, but it's still a real GPU result. Hand it to Caterpillar, which is built for CPU and shared graphics. What it deliberately will not do is quietly pin an oversized model into shared memory and let it swap — that produces a machine that feels broken for no visible reason. If your model is too big for comfort, Lizard now says so and picks a supported path instead. Rule of thumb: a Q4 model needs roughly 60% of the download size in free graphics memory to stay fully resident. Anyone running a large model on a small card? What did you settle on?

Inference Engineering · Day 5 · Evening [image: Screenshot%202026-07-22%20214012.png] What if one pass over the model weights could accept several tokens? Caterpillar's speculative path drafts a short sequence, then verifies multiple candidates in a single weight pass. It is lossless: rejected drafts fall back to the normal path, so accepted output remains the output the model would have produced. On predictable or repetitive text, our tests reached up to 2× speed. On ordinary prose, the path steps aside rather than forcing a bad optimization. Optimization should be conditional when the workload is conditional. Engineering fact: Caterpillar's lossless speculative path can verify multiple drafted tokens per weight pass on predictable output and steps aside on ordinary prose. Read the relevant Lizard page #SpeculativeDecoding #CaterpillarEngine #LLMInference #LocalAI #LizardLLM <!-- lizard-marketing-slot:day-05-pm -->

Inference Engineering · Day 5 · Morning [image: day-05-am-universal-1785203607511.png] Caterpillar now defaults to physical cores, and that change came from a simple result: on a four-core test laptop, using every logical thread made decode slower. Switching to the physical-core count improved measured decode by about 9% and also made the timing less noisy. The important part is not the exact laptop. It is the shape of the workload. Decode can be limited by memory bandwidth long before it is limited by raw thread count. Once that happens, hyperthreads stop looking like extra capacity and start looking like contention. More schedulable threads can mean more pressure on the same caches, the same memory channels, and the same execution resources. That is why the default changed. A local runtime should choose the thread count that matches the bottleneck, not the number that looks best on a spec sheet. The override still matters for experiments and unusual machines, but the default should bias toward the setting that is most likely to behave well without tuning. I have seen this pattern enough times to treat thread count as part of measurement, not just configuration. If a workload is bandwidth-bound, the “more threads” instinct can be the wrong first move. What’s the first sign you use to tell whether extra logical threads are helping, or just adding noise? Engineering fact: Caterpillar defaults to physical cores; on a four-core test laptop this improved decode by about 9% versus using every logical thread. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #cpu #performance #benchmarking #systems <!-- lizard-marketing-slot:day-05-am -->

Inference Engineering · Day 4 · Evening [image: day-04-pm-universal-1785125811157.png] The Q4 change was a layout rewrite, not a new decode path. Caterpillar's four-bit weights were repacked around eight-row tiles, and the scale data now gets unpacked once at load instead of being pulled back through the hot loop on every token. That sounds like a small structural change, but on test hardware it cut bytes read per token by about 13% and improved decode by about 8% while keeping the output byte-identical. The practical lesson is simple: once decode is memory-bound, the difference between “prepare it once” and “touch it every token” starts to show up as throughput. The math did not change. The answer did not change. The memory traffic did. That is the part worth watching in your own kernels and data paths. If a format change makes the hot loop carry less baggage, you often get the speedup without any behavioral risk. Have you seen a layout change beat a kernel rewrite on a workload you care about? Engineering fact: On Lizard's test hardware, Caterpillar's eight-row Q4 layout reduced bytes read per token by about 13% and improved decode by about 8% with byte-identical output. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #Caterpillar #Q4 #LLMInference #SystemsProgramming #PerformanceEngineering <!-- lizard-marketing-slot:day-04-pm -->