Inference Engineering · Day 25 · Evening [image: two-engines-decision.png] A runtime choice should survive beyond the activation form. Lizard records the selected native provider with the active model. Onboarding warmup and Chat resolve that state rather than reverting to an unrelated default after the page changes. Persistent provider identity makes performance evidence and troubleshooting comparable across surfaces. Where in your workflow could a provider choice be silently lost? Engineering fact: When a local model is activated, Lizard stores whether lizard-native or Caterpillar was selected and uses that provider for onboarding warmup and Chat. Read the relevant Lizard page #Caterpillar #LizardLLM #GPUProgramming #LocalAI #GGUF <!-- lizard-marketing-slot:day-25-pm -->

Inference Engineering · Day 18 · Evening [image: two-engines-decision.png] Caterpillar is an inference runtime, not another model or agent orchestrator. It loads the same GGUF foundation but organizes execution around a precompiled compute-graph plan. The design target is less repeated host coordination around token generation, with its own shader cache and optimized paths. Keeping it separate from lizard-native lets both engines evolve and remain measurable. Which part of your token loop is still being rebuilt or resubmitted every step? Engineering fact: Caterpillar is a separate GGUF runtime whose execution core uses a precompiled compute-graph plan to reduce repeated CPU-to-GPU coordination around token generation. Read the relevant Lizard page #Caterpillar #LizardLLM #GPUProgramming #LocalAI #GGUF <!-- lizard-marketing-slot:day-18-pm -->

Inference Engineering · Day 18 · Morning [image: two-engines-decision.png] A provider selector is meaningless if the runtime silently launches a different engine. Lizard carries the requested provider through server launch, health checks, benchmark artifacts, onboarding activation, and Chat telemetry. A mismatch is treated as an error rather than a successful run under the wrong label. Provider identity belongs in both execution and evidence. Can you prove which engine served the last response in your local stack? Engineering fact: Lizard exposes lizard-native and Caterpillar as separate runtime providers in serve, onboarding, benchmark, and Chat paths instead of silently substituting one for the other. Read the relevant Lizard page #Caterpillar #LizardLLM #GPUProgramming #LocalAI #GGUF <!-- lizard-marketing-slot:day-18-am -->

Inference Engineering · Day 7 · Evening [image: Screenshot%202026-07-22%20213232.png] We implemented an AVX-512 VNNI decode kernel and chose not to enable it by default. The instruction is promising on supported Intel and AMD CPUs. But on the hardware available for validation, the measured result remained inside ordinary run-to-run noise. Caterpillar ships the proven path as default and keeps VNNI behind an explicit experiment flag. Engineering credibility sometimes means declining to claim a speedup. Engineering fact: Caterpillar's AVX-512 VNNI kernel remains opt-in because it measured within normal run-to-run noise on available test hardware. Read the relevant Lizard page #AVX512 #CPUOptimization #Benchmarking #CaterpillarEngine #LizardLLM <!-- lizard-marketing-slot:day-07-pm -->

Inference Engineering · Day 5 · Evening [image: Screenshot%202026-07-22%20214012.png] What if one pass over the model weights could accept several tokens? Caterpillar's speculative path drafts a short sequence, then verifies multiple candidates in a single weight pass. It is lossless: rejected drafts fall back to the normal path, so accepted output remains the output the model would have produced. On predictable or repetitive text, our tests reached up to 2× speed. On ordinary prose, the path steps aside rather than forcing a bad optimization. Optimization should be conditional when the workload is conditional. Engineering fact: Caterpillar's lossless speculative path can verify multiple drafted tokens per weight pass on predictable output and steps aside on ordinary prose. Read the relevant Lizard page #SpeculativeDecoding #CaterpillarEngine #LLMInference #LocalAI #LizardLLM <!-- lizard-marketing-slot:day-05-pm -->

Inference Engineering · Day 5 · Morning [image: day-05-am-universal-1785203607511.png] Caterpillar now defaults to physical cores, and that change came from a simple result: on a four-core test laptop, using every logical thread made decode slower. Switching to the physical-core count improved measured decode by about 9% and also made the timing less noisy. The important part is not the exact laptop. It is the shape of the workload. Decode can be limited by memory bandwidth long before it is limited by raw thread count. Once that happens, hyperthreads stop looking like extra capacity and start looking like contention. More schedulable threads can mean more pressure on the same caches, the same memory channels, and the same execution resources. That is why the default changed. A local runtime should choose the thread count that matches the bottleneck, not the number that looks best on a spec sheet. The override still matters for experiments and unusual machines, but the default should bias toward the setting that is most likely to behave well without tuning. I have seen this pattern enough times to treat thread count as part of measurement, not just configuration. If a workload is bandwidth-bound, the “more threads” instinct can be the wrong first move. What’s the first sign you use to tell whether extra logical threads are helping, or just adding noise? Engineering fact: Caterpillar defaults to physical cores; on a four-core test laptop this improved decode by about 9% versus using every logical thread. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #cpu #performance #benchmarking #systems <!-- lizard-marketing-slot:day-05-am -->

Inference Engineering · Day 4 · Evening [image: day-04-pm-universal-1785125811157.png] The Q4 change was a layout rewrite, not a new decode path. Caterpillar's four-bit weights were repacked around eight-row tiles, and the scale data now gets unpacked once at load instead of being pulled back through the hot loop on every token. That sounds like a small structural change, but on test hardware it cut bytes read per token by about 13% and improved decode by about 8% while keeping the output byte-identical. The practical lesson is simple: once decode is memory-bound, the difference between “prepare it once” and “touch it every token” starts to show up as throughput. The math did not change. The answer did not change. The memory traffic did. That is the part worth watching in your own kernels and data paths. If a format change makes the hot loop carry less baggage, you often get the speedup without any behavioral risk. Have you seen a layout change beat a kernel rewrite on a workload you care about? Engineering fact: On Lizard's test hardware, Caterpillar's eight-row Q4 layout reduced bytes read per token by about 13% and improved decode by about 8% with byte-identical output. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #Caterpillar #Q4 #LLMInference #SystemsProgramming #PerformanceEngineering <!-- lizard-marketing-slot:day-04-pm -->

Inference Engineering · Day 3 · Evening [image: day-03-pm-universal-1785066334537.png] If you trace token latency in a local runtime, the obvious place to look is often the wrong one. The model kernels can look healthy while the loop still loses time to coordination: CPU work sets up the next step, the GPU waits for a handoff, and the runtime repeats the same planning work for every token. That kind of overhead is small in isolation, but over a long decode it becomes part of the profile you actually feel. Caterpillar changes that by rebuilding the execution core around a pre-compiled compute-graph plan. The useful effect is not just that the graph exists. It is that the runtime can stop reconstructing the same execution shape on every token and instead follow a plan that is already laid out. That reduces per-token CPUGPU synchronization, which is exactly the sort of cost that hides between kernel launches. The practical lesson is to separate math cost from orchestration cost. When I look at a slow generation path, I want to know whether the runtime is paying for repeated scheduling, repeated graph setup, or repeated handoff boundaries that could have been planned once. If the execution shape is stable, the hot path gets simpler. If it is not, even good kernels can end up carrying latency that never shows up in FLOPS. One thing I have found useful is to inspect the decode path at the token boundary, not just the kernel level. If the coordination layer is doing more work than expected, the fix is usually structural rather than numerical. Have you seen a case where reducing CPUGPU synchronization moved token latency more than changing the kernel itself? Engineering fact: Caterpillar rebuilds the execution core around a pre-compiled compute-graph plan to reduce per-token CPUGPU synchronization. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #Caterpillar #ComputeGraph #Inference #WindowsAI #LocalLLM <!-- lizard-marketing-slot:day-03-pm -->

Inference Engineering · Day 3 · Morning [image: day-03-am-universal-1785057232972.png] A startup trace is easier to trust when one expensive step stops repeating. Shader compilation is a good example. If the first launch is slow, that is expected on some GPU paths. The more useful question is whether later launches are still paying the same warm-up cost. When that happens, the slowdown is not really about inference speed. It is repeated setup work hiding in the launch path. Both native engines now persist compiled GPU shaders, so the first warm-up is paid once instead of on every load. That changes how I read the trace. A cold start can still be expensive, but a warm start should converge. If it does not, I want to look at cache persistence, residency decisions, and any path that forces the runtime to rebuild GPU work before it reaches the model. In practice I split startup into three buckets: cold load, warm load, and steady-state decode. Cold load tells me what the system pays to become ready. Warm load tells me whether setup work is being repeated. Steady-state decode is where model-path changes should show up clearly. If those three are mixed together, you end up tuning the wrong layer and the numbers stay noisy. That is why this small persistence change matters. It makes startup behavior more legible. A trace that used to look like a slow model can now point to the real culprit: repeated shader compilation, repeated cache rebuilds, or a launch sequence that is doing more work than it should. When you profile GPU startup, what is the first thing you separate: cold load, warm load, or decode? Engineering fact: Both native engines persist compiled GPU shaders so the first warm-up is paid once instead of on every load. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #GPUShaders #ShaderCompilation #StartupLatency #NativeRuntime #PerformanceDebugging <!-- lizard-marketing-slot:day-03-am -->

Caterpillar is a separate clean-room inference provider. Its design makes graph validation, temporary-memory reuse, and execution planning explicit before token-by-token replay. [image: caterpillar-graph-plan.webp] From GGUF to token stream Own GGUF reader — metadata and tensors are read by Caterpillar's standalone loader. Typed DAG validation — shapes and dependencies are checked before execution. Lifetime-reused activation arena — temporary storage can be reused when tensor lifetimes do not overlap. Precompiled graph plan — repeatable work is represented as a plan instead of rediscovered blindly for every decoded token. Coalesced CPU spans — compatible CPU operations can replay in grouped spans; supported paths can add speculative draft and verification. Caterpillar does not include llama.cpp, ggml, ik_llama, or Lizard Native source units. The providers can be measured beside each other, but they are not the same implementation. The engine is actively evolving, so supported native combinations are narrower than the compatibility paths. The landing page names that boundary instead of turning planned coverage into a present-tense claim. Read the complete layer map and comparison: https://lizard-llm.qendryx.com/technical-overview.html Question: Which part should we publish next in more depth: graph validation, activation lifetimes, compiled spans, or speculative verification?

For anyone who likes the details. Modern Intel (Ice Lake and newer) and AMD (Zen 4 and newer) processors have an instruction that does 8-bit integer maths faster. Caterpillar has a kernel that uses it, behind CATERPILLAR_CPU_VNNI=1. It ships disabled, and the reason is worth stating plainly: on our test hardware it measured within run-to-run noise of the standard path. Not faster, not slower — indistinguishable. Shipping it on by default would be claiming a win we can't demonstrate. It may well help on other chips, particularly ones with more memory bandwidth. If you try it: CATERPILLAR_CPU_VNNI=1 Please post your CPU and both numbers, run back-to-back. Alternating measurements matter here — machines get hotter as you test them, and that alone can swamp the difference.

Short answer: leave it alone — the default is now your physical core count, which measured fastest. Longer answer, because it surprises people: using every logical thread is usually slower, not faster. On a 4-core/8-thread laptop, 4 threads measured about 9% faster than 8, with steadier timings. The reason is that generating text is limited by how fast the machine can read the model from memory, not by how much arithmetic it can do. Extra threads add contention without adding memory bandwidth. If you want to experiment, CATERPILLAR_CPU_THREADS overrides it. If you try it, please post both numbers and your CPU — one result on its own is hard to learn from, since thermal state moves these numbers around a lot.

You may notice Caterpillar getting noticeably quicker on some kinds of output — lists, structured text, anything repetitive — and behaving normally on ordinary prose. That's deliberate. The short version: generating a word normally means reading the whole model from memory once. If the next few words are predictable, Caterpillar drafts a guess and checks several words in a single pass instead of one pass each. Two things worth knowing: It is lossless. You get exactly the same words you would have got anyway — it's verified against the real model, not an approximation. If a guess is wrong it's thrown away. It steps aside when it isn't helping. On normal prose, guesses rarely land, so it stops trying rather than making you pay for failed guesses. So the honest summary is: sometimes much faster, never slower in a way you'd notice, never different output. Anyone seen a big speed-up on a particular kind of prompt?

Caterpillar is the newer of Lizard's two engines. It exists because a gaming PC and a work laptop need genuinely different approaches to run the same model well. Where lizard-native leans on your graphics card, Caterpillar is built to get the most out of: your processor, when there's no dedicated card, and shared/built-in graphics, where memory is tight. Lizard picks it automatically — you never choose. It takes over when there's no discrete GPU, or when the model is too large to sit on the one you have. It's also where most active development happens right now. Recent work made CPU decoding meaningfully faster and cut how much memory it reads for each word it generates. More detail: https://lizard-llm.qendryx.com/product.html If you're on a laptop with no dedicated card, this is the engine you're using — how does it feel?