Lizard Native

Lizard inference engineering: Context length belongs in lived performance history

lizardadmin — Wed, 19 Aug 2026 11:00:13 GMT

Inference Engineering · Day 27 · Evening [image: lizard-native-stack.png] Conversation length changes the workload before decode begins. As context grows, prompt processing and KV-cache pressure can change even when the model and hardware stay the same. Lizard records context tokens and includes a context bucket in comparable turn history. That keeps a long-conversation slowdown from looking like an unexplained engine regression. At what context length does your current local model begin to feel different? Engineering fact: Lizard records context tokens and groups native turn history into context buckets because longer conversations change prefill and KV-cache costs. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Chat metrics should be scoped to the active provider

lizardadmin — Wed, 19 Aug 2026 01:00:11 GMT

Inference Engineering · Day 27 · Morning [image: lizard-native-stack.png] A performance history loses meaning when different engines share one average. Lizard records the runtime provider on every turn and exposes per-provider summaries in Chat. That keeps lizard-native, Caterpillar, llama.cpp, and Ollama histories separate when the user compares median, best, and latest behavior. The metric should follow the engine that actually served the answer. Would you notice if your chat application switched runtimes halfway through a test? Engineering fact: Lizard's workspace Chat groups native turn history by runtime provider so lizard-native, Caterpillar, llama.cpp, and Ollama results are not merged into one misleading average. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Prompt caching should follow conversation structure

lizardadmin — Thu, 13 Aug 2026 11:00:10 GMT

Inference Engineering · Day 21 · Evening [image: lizard-native-stack.png] The same cache can be a product feature and a benchmark contaminant. A conversation naturally repeats its previous prefix, so retaining prompt state can avoid reprocessing the whole transcript. A benchmark that repeats an identical prompt for measurement should not accidentally turn that reuse into a fake prefill win. Cache policy should follow workload semantics, not simply remain globally enabled. Which caches in your stack need different rules for production and benchmarking? Engineering fact: Lizard Chat can reuse prior prompt state for the same conversation while benchmark paths avoid chat-only cache behavior that would make repeated prompts look artificially cheap. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Warm the selected model before the first conversation

lizardadmin — Thu, 13 Aug 2026 01:00:09 GMT

Inference Engineering · Day 21 · Morning [image: lizard-native-stack.png] Cold loading and conversation latency are different product moments. Lizard can preload the selected native model during onboarding or Chat warmup. The model becomes resident before the first real prompt, while the interface reports load state rather than presenting a long silent wait as generation. Separating readiness from response time makes both the product and the benchmark easier to understand. Where in your workflow should model warmup happen? Engineering fact: Lizard onboarding and Chat can preload the selected native model so the first user message does not have to absorb the entire cold model-load cost. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: CPU-only should remain a first-class path

lizardadmin — Tue, 11 Aug 2026 11:00:09 GMT

Inference Engineering · Day 19 · Evening [image: lizard-native-stack.png] A local runtime still needs a truthful path when no compatible GPU backend is available. Lizard can execute supported GGUF graphs on CPU and records the backend decision. Hardware capability, model support, and memory evidence determine whether GPU, hybrid, or CPU execution is appropriate. A slower supported path is better than an opaque fallback pretending to be GPU acceleration. Does your runtime expose when a request actually fell back to CPU? Engineering fact: Lizard retains a native CPU execution path and uses hardware capability and memory evidence to decide when GPU or hybrid execution is unavailable or inappropriate. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Integrated GPUs need a shared-memory plan

lizardadmin — Tue, 11 Aug 2026 01:00:07 GMT

Inference Engineering · Day 19 · Morning [image: lizard-native-stack.png] Integrated GPUs change the memory question, not just the adapter name. On a shared-memory system, the model, KV cache, application, compositor, and CPU all compete inside one physical budget. Lizard distinguishes this configuration when planning fit and execution rather than treating advertised graphics memory as an isolated pool. The useful number is the memory currently available to the whole workload. How does your runtime budget an iGPU while the desktop and other applications are active? Engineering fact: Lizard's hardware planning distinguishes integrated-memory and discrete-memory systems so model fit and execution choices can account for shared RAM instead of treating every GPU like a separate VRAM device. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Capacity still matters when no benchmark exists

lizardadmin — Fri, 07 Aug 2026 11:00:08 GMT

Inference Engineering · Day 15 · Evening [image: lizard-native-stack.png] A new model still needs a safe first serving decision. When no matching HTTP profile exists, Lizard estimates the eligible lane from the KV plan, available memory, and any operator ceiling. The result is labeled as capacity policy, not measured performance. Safe defaults and measured optimization can coexist if their provenance remains visible. Does your runtime tell you whether a concurrency choice was measured or inferred? Engineering fact: When matching HTTP evidence is unavailable, Lizard chooses a concurrency lane from KV capacity, available memory, and any explicit ceiling rather than fabricating a measured result. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Let measured B16 and B8 results choose the lane

lizardadmin — Thu, 06 Aug 2026 01:00:12 GMT

Inference Engineering · Day 14 · Morning [image: lizard-native-stack.png] A fixed concurrency default ignores both hardware capacity and measured behavior. Lizard treats B=16 as the throughput target and B=8 as the normal reduced-capacity lane. When both have completed HTTP evidence, the faster eligible primary lane wins. B=4 and B=1 are reserved for constrained fallback. The policy turns a benchmark result into a serving decision without pretending every machine has the same memory budget. Would your runtime choose the same batch lane on an integrated GPU and a discrete GPU? Engineering fact: Lizard's adaptive concurrency policy chooses the faster completed B=16 or B=8 HTTP measurement when capacity permits; B=4 and B=1 remain fallbacks. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: TTFT and completion time diagnose different pain

lizardadmin — Wed, 05 Aug 2026 10:27:08 GMT

Inference Engineering · Day 13 · Evening [image: lizard-native-stack.png] One latency number is too blunt for streamed output. Time to first content tells you when the answer starts. End-to-end response time tells you when it is actually done. Latency per generated token adds a third view: it normalizes a reply so a long completion does not hide a slow decode path behind a fast start. That separation matters in practice. A slow first token can point to loading, prompt setup, or transport. A slow finish with a decent first token usually shifts attention to decode, token pacing, or work that accumulates after the stream has already begun. If you compress both into one figure, you lose the shape of the delay and the place to look next. Lizard records time to first content, end-to-end response time, and latency per generated token on native HTTP and Chat turns, so those cases stay visible instead of being averaged away. That makes it easier to compare a request that feels responsive at the start with one that reaches the user too late. In a real system, which symptom tends to show up more often for you: slow first content or slow completion? Engineering fact: Lizard records time to first content, end-to-end response time, and latency per generated token for native HTTP and Chat turns. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #LLMObservability #Latency #StreamingAI #DeveloperTools #Performance

Lizard inference engineering: Separate user-visible speed from native decode speed

lizardadmin — Wed, 05 Aug 2026 00:39:09 GMT

Inference Engineering · Day 13 · Morning [image: lizard-native-stack.png] Decode speed and user-visible speed answer different operational questions, and serving stacks get into trouble when they are treated as the same metric. Lizard Chat and native HTTP responses keep both numbers in view: the worker’s decode-only token rate, and the end-to-end rate the user actually experiences. That split matters because prompt handling, scheduling, transport, and response assembly all happen outside the decode loop. If you only watch decode tok/s, it is easy to miss where latency is really being spent. In practice, the decode number is still useful. It tells you whether the kernel is making progress once the worker is running. But the user-visible number is the one that reflects the full request path. When those two move together, the serving path is usually healthy. When they diverge, the gap is a clue: session setup may be too expensive, prefill may be creeping up, or the handoff between worker and client may be doing more work than expected. That distinction is especially useful in local runtimes, where overheads can dominate short requests. A model can look fast in isolation and still feel sluggish in chat because the request spends too much time before the first token or after the last one. Separating the metrics makes that visible without arguing over which number is “real.” I’d rather see both than compress them into one optimistic line item. Which part of your serving path only showed up after you split decode tok/s from user-visible tok/s? Engineering fact: Lizard Chat and native HTTP responses track end-to-end user-visible token rate separately from the worker's decode-only token rate. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #LocalAI #LLMOps #Inference #TokenThroughput #SystemsEngineering

Lizard inference engineering: Make KV-cache precision visible

lizardadmin — Thu, 30 Jul 2026 01:00:13 GMT

Inference Engineering · Day 7 · Morning [image: lizard-native-stack.png] KV-cache is one of those pieces that only looks invisible until a long chat starts crowding out memory. Each generated token adds keys and values that have to stay around for later attention, so cache precision is not a trivia setting. It is part of the capacity plan. lizard-native makes that choice explicit with f16, q8_0, and q4_0 KV-cache precision. That matters because the trade is real: preserve more fidelity, or fit more context inside the same machine budget. When the decision is hidden, people tend to blame the model when the actual limit is memory. The useful part is not that one setting is always best. It is that the runtime surfaces the cost where the user can reason about it. On a laptop or a small workstation, that changes the conversation from “why did this stop?” to “what do I want to spend memory on?” In practice, I would rather see a clear cache trade than a vague “supported” label. It gives operators something they can tune against the session length they actually need, instead of discovering the limit after the context has already grown. When you are sizing local inference for real users, do you bias toward higher cache precision or more room for conversation length? Engineering fact: lizard-native exposes f16, q8_0, and q4_0 KV-cache precision as an explicit context-length versus memory trade. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #KVCache #ContextWindow #LocalInference #SystemsEngineering #WindowsAI

Lizard inference engineering: Use UMA without a fake copy

lizardadmin — Wed, 29 Jul 2026 11:00:12 GMT

Inference Engineering · Day 6 · Evening [image: lizard-native-stack.png] An integrated GPU is not a small discrete GPU. On unified-memory hardware, CPU and GPU already access the same physical memory. Treating that system like a discrete card can add unnecessary staging and copies. lizard-native includes a UMA zero-copy path for supported integrated graphics. The design goal is simple: shared weights should stay shared, while the Direct3D 12 execution path remains explicit. Hardware-aware inference starts by respecting the memory architecture you actually have. Engineering fact: lizard-native includes a zero-copy path for unified-memory integrated GPUs so shared memory is treated as shared rather than copied as if discrete. Read the relevant Lizard page #LizardNative #IntegratedGPU #D3D12 #ZeroCopy #LocalAI

Lizard inference engineering: Stop rebuilding descriptors per token

lizardadmin — Wed, 29 Jul 2026 01:00:14 GMT

Inference Engineering · Day 6 · Morning [image: Screenshot%202026-07-22%20222539.png] A decode loop can look GPU-bound and still waste time on the host. In the lizard-native Gemma path, roughly 400 weight descriptors were being rebuilt for every generated token. That meant the runtime kept reconstructing the same per-token metadata before the next step could move forward. The GPU had work to do, but so did the scaffolding around it, and that scaffolding sat on the critical path. The fix was to stop rebuilding those descriptors and keep them around for reuse. On the Gemma test runs, that improved decode by 13–26% and brought first-token response back by about 20%. The useful lesson is that decode tuning is rarely only about the kernel. If the loop repeats cleanly, inspect the state that gets recreated every pass: descriptor setup, graph metadata, plan assembly, session bookkeeping, cache wiring. Those are easy to ignore because they do not look like compute, but they can still decide how fast the next token appears. I have learned to ask a boring question before touching the math: what is being rebuilt on every token that could have been carried forward from the previous one? In practice, persistent per-token state often buys a cleaner win than shaving a few cycles off the math itself. Less allocation, less synchronization pressure, fewer host-side stalls, and a shorter path back into the decode loop. When the hot path is stable, that kind of cleanup tends to show up immediately in the first-token feel as well as steady-state throughput. Engineering fact: The lizard-native Gemma path stopped rebuilding roughly 400 weight descriptors per token; test runs improved decode 13–26% and first-token response about 20%. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #inferencetuning #decodeperformance #windowsai #d3d12

Lizard Native benchmark CLI and HTTP command reference

lizardadmin — Tue, 28 Jul 2026 16:30:41 GMT

Lizard Native benchmark CLI and HTTP guide Run the user-visible OpenAI-compatible serve path against a stock llama.cpp server with the same Llama 3.2 GGUF: npm run bench:native:http -- ` --provider lizard-native ` --model llama-3.2-3b-instruct ` --gguf "$env:USERPROFILE\.lizard-llm\models\llama-3.2-3b-instruct.gguf" ` --batch "16,8,4,1" ` --tokens 64 ` --label lizard-native-llama32-same-weights-http Use --skip-llama for provider-only tuning. That mode produces no admissible cross-runtime ratio. Adaptive concurrency The native policy uses B=16 as the aggregate-throughput target and B=8 as the normal secondary lane. B=4 and B=1 are used only when both primary lanes are unavailable. Automatic result selection chooses the faster successful B=16/B=8 measurement; a failed or token-unmeasured lane is never promoted. What a valid llama.cpp ratio requires The candidate and control must load the same physical GGUF file. Keep the model, quantization, prompt, output token budget, concurrency, warm-up and machine conditions fixed. A Llama 3.2 candidate cannot be compared with a Gemma control. The HTTP harness refuses a mismatched canonical path. The artifact records provider identity, candidate/control paths, completion and failure counts, aggregate completion tokens per second, full-response latency, lane selection and provenance. The current non-streaming harness does not claim TTFT or TPOT. Read the complete command reference The public guide includes copy buttons, parameters, output interpretation, throughput and latency runs, serving, offline and online checks, repeated-prefix workloads, sequence-length and stress runs, production checks, and current multi-GPU/tensor-parallel/data-parallel limitations: https://lizard-llm.qendryx.com/benchmark-cli.html Benchmark methodology and screenshots: https://lizard-llm.qendryx.com/benchmarks.html Technical architecture: https://lizard-llm.qendryx.com/technical-overview.html

Lizard inference engineering: Stop oversized models from thrashing

lizardadmin — Sat, 25 Jul 2026 11:00:06 GMT

Inference Engineering · Day 2 · Evening A model that almost fits can create the worst failure mode: it runs, but it thrashes. On a tight GPU budget, repeatedly evicting and re-uploading layers can reduce decode to a crawl. A single “GPU or CPU” decision is too coarse. Lizard plans prefill and decode independently. It can keep GPU-assisted prompt processing while moving decode to a bounded, stable path when full residency is unsafe. The engineering lesson: split the phases before you split the blame. Engineering fact: When a model exceeds the usable GPU weight budget, Lizard can keep GPU-assisted prompt processing and use bounded hybrid decode instead of repeated eviction. Read the relevant Lizard page #VRAM #LLMInference #GPUPerformance #LizardNative #LocalAI

Lizard inference engineering: Budget against free VRAM, not the sticker

lizardadmin — Sat, 25 Jul 2026 01:00:11 GMT

Inference Engineering · Day 2 · Morning The number on the GPU box is not your inference budget. Windows, the display pipeline, browsers and other applications already occupy graphics memory. Planning against total VRAM can turn a model that looks safe into a load failure or an eviction loop. Lizard asks the driver what is available now, then budgets model weights and runtime needs against that live figure. If the fit is unsafe, it chooses a supported path and reports why. Capacity planning should begin with free memory, not advertised memory. Engineering fact: Lizard checks the graphics driver's currently available memory rather than assuming the GPU's advertised total is free. Read the relevant Lizard page #VRAM #GPUEngineering #LocalAI #LizardLLM #InferenceEngineering

Lizard inference engineering: Keep model weights resident

lizardadmin — Fri, 24 Jul 2026 11:00:07 GMT

Inference Engineering · Day 1 · Evening Move tokens—not the model. A common local-inference cost is repeated setup and data movement. lizard-native takes a resident-weight approach: load the GGUF weights onto the Direct3D 12 device once, keep them there across calls, and reuse the warmed execution path. That does not make memory limits disappear. It makes residency an explicit planning decision instead of an accidental runtime surprise. The useful question becomes: does the model, cache and current workload fit safely on this device? Engineering fact: lizard-native uploads model weights once and keeps them resident on a Direct3D 12 GPU across calls. Read the relevant Lizard page #LizardNative #D3D12 #GPUInference #LocalAI #LizardLLM

Technical overview: how Lizard Native keeps Direct3D 12 inference resident

lizardadmin — Thu, 23 Jul 2026 05:42:14 GMT

Lizard Native is Lizard's stable, Windows-first native provider. For a supported model, it uploads quantized tensors to GPU-resident buffers and executes decode through Direct3D 12 compute shaders. The model-fit decision happens before execution and uses the exact GGUF identity plus the machine's available memory. [image: lizard-native-architecture.webp] The layers Hardware and model-fit planning — CPU, memory, Direct3D 12 capabilities, model family, quantization, and live budget are evaluated before a native lane is selected. Resident model tensors — supported quantized weights stay in device buffers while the model is active instead of being reloaded for every request. Compute shaders — quantized operations run through Direct3D 12 compute; compiled shader artifacts are cached on disk. Memory discipline — residency is budgeted, and compatible UMA hardware can use a shared-memory path. Provider identity — a native row stays labeled Lizard Native. A llama.cpp or Ollama fallback is not reported as native. The boundary matters Lizard Native implements a tested native subset; it does not claim that every GGUF architecture and quantization runs on this provider. Unsupported combinations stay eligible for an explicit bundled llama.cpp or optional Ollama fallback. Read the complete layer map and comparison: https://lizard-llm.qendryx.com/technical-overview.html Question: On your Windows hardware, is the limiting factor available memory, shader execution, model coverage, or per-token scheduling?

Which graphics cards work — post yours

lizardadmin — Sun, 19 Jul 2026 10:34:00 GMT

lizard-native needs a DirectX 12 capable adapter. In practice that's essentially anything from the last several years, including integrated graphics. Confirmed working so far: Intel Iris Xe (built into many laptops) — works well, memory footprint matters most here NVIDIA and AMD discrete cards — including the case where the model is larger than the card If you'd like to add yours, this format is genuinely useful to others: GPU: e.g. RTX 4060 8GB / Iris Xe Model + size: e.g. llama-3.2-3b Q4_K_M Engine chosen: lizard-native or caterpillar Tokens/sec: from the readiness check You can read tokens/sec straight off the readiness check in setup, no benchmark credits needed.

Running a model that's bigger than your graphics card

lizardadmin — Thu, 16 Jul 2026 10:25:00 GMT

You don't need a card big enough to hold the whole model. When a model doesn't fit, Lizard has two honest options and it will tell you which one it took: Stream the weights through the GPU. The card still does the work, but it reads parts of the model as it goes. Slower than fully resident, but it's still a real GPU result. Hand it to Caterpillar, which is built for CPU and shared graphics. What it deliberately will not do is quietly pin an oversized model into shared memory and let it swap — that produces a machine that feels broken for no visible reason. If your model is too big for comfort, Lizard now says so and picks a supported path instead. Rule of thumb: a Q4 model needs roughly 60% of the download size in free graphics memory to stay fully resident. Anyone running a large model on a small card? What did you settle on?

Why the first load is slow and every one after is fast

lizardadmin — Mon, 13 Jul 2026 10:16:00 GMT

A question that comes up a lot: "the first time I load a model it takes a while, then it's instant. Why?" The engine compiles small GPU programs (shaders) for your specific hardware the first time it runs. That compile used to happen on every load, which meant waiting a minute or more each time. Those compiled shaders are now written to disk and reused. So: First ever load on this machine: a few seconds of compiling. Every load after: near-instant, including different models and after a reboot. If your first load is still slow every single time, that's worth reporting — it usually means the cache directory isn't writable. How long does a load take on your setup, cold vs. warm?

What lizard-native is, and when Lizard picks it

lizardadmin — Sat, 11 Jul 2026 10:50:00 GMT

lizard-native is the engine that runs your model on your graphics card. It loads the model's weights into GPU memory once and keeps them there. That's the whole trick: the model stays resident, so after the first load, answers start almost immediately instead of re-reading gigabytes from disk every time. Lizard chooses it automatically when: you have a DirectX 12 graphics card (including built-in Intel Iris Xe), and the model actually fits in that card's memory alongside everything else running. What you'll notice: First load takes a few seconds while shaders compile. That result is cached on disk, so every load after is near-instant — even for a different model, or after restarting. Memory use is lower than you'd expect for the file size, because of how the weights are packed. If it isn't being used, it's usually one of two things: no DirectX 12 adapter, or the model is bigger than the card. Lizard then hands the work to Caterpillar instead — that's by design, not a failure. More detail: https://lizard-llm.qendryx.com/product.html What card are you running, and what does your first-load time look like?