Inference Engineering · Day 10 · Evening

Ten days of inference engineering ends with the most important benchmark input: your question.

Our published results belong to the hardware, model, quantization, prompt and token budget shown. They identify real bottlenecks; they do not create a universal provider ranking.

What should we test next?

Please include your CPU/GPU, available memory, model, quantization and the decision you need the benchmark to answer.

Engineering fact: Published Lizard results are model-, quantization-, prompt-, hardware-, and run-specific; the next useful test should be selected with that context.

Read the relevant Lizard page

#Benchmarking #LocalAI #InferenceEngineering #LizardNative #CaterpillarEngine

<!-- lizard-marketing-slot:day-10-pm -->

Inference Engineering · Day 10 · Morning

If you are trying to keep a local runtime easy to adopt, the API shape matters as much as the model choice.

After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint. That means an existing app can usually keep its client logic, swap the base URL, and keep moving. You still get local prompts and outputs, which is the part that usually decides whether a team treats a runtime as a tool or as a project.

That compatibility also changes how you test integrations. You can validate request format, retry behavior, and response handling against a local endpoint before you ever point anything at a remote provider. For Windows teams especially, that is a practical way to reduce friction without giving up the familiar /v1/chat/completions style contract.

The lesson is simple: local-first works better when it changes the data path, not the application’s expectations. If the endpoint stays familiar, the migration is mostly about deployment and model choice instead of rewriting every client.

When you’ve brought a local model endpoint into an existing app, what usually breaks first: auth assumptions, streaming, or tool-calling shape?

Engineering fact: After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint while prompts and outputs stay on the machine.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#localai #openaicompatible #windows #llmops #apidesign

<!-- lizard-marketing-slot:day-10-am -->

Inference Engineering · Day 9 · Evening

A benchmark dashboard can mislead without containing a false number.

Treat an excluded provider as zero. Mix warm-up into decode. Turn one metric winner into a universal recommendation. Each shortcut changes the story.

Lizard keeps provider identity, wall time, end-to-end throughput, native decode, load, warm-up and memory separate. Missing providers stay unmeasured.

The goal is not to remove judgment. It is to keep the evidence needed for judgment.

Engineering fact: Lizard's dashboard keeps excluded providers unmeasured and separates wall time, end-to-end throughput, decode, warm-up, and memory.

Read the relevant Lizard page

#Benchmarking #DataIntegrity #LLMInference #Reproducibility #LocalAI

<!-- lizard-marketing-slot:day-09-pm -->

Inference Engineering · Day 9 · Morning

A signed GPU result must prove that the GPU path actually ran.

When a remote benchmark model is larger than a discrete card's memory, a quiet CPU fallback would produce a number—but not the number requested.

lizard-native can instead stream weights through the Direct3D 12 decode lane. It is slower than full residency, and the result says so, but the measurement remains a genuine DirectX path with adapter and backend evidence.

A slower honest result is more useful than a faster mislabeled one.

Engineering fact: For remote benchmarks on a discrete GPU smaller than the model, lizard-native can stream weights through the D3D12 lane rather than silently reporting a CPU fallback as GPU.

Read the relevant Lizard page

#D3D12 #GPUInference #Benchmarking #LizardNative #LocalAI

<!-- lizard-marketing-slot:day-09-am -->

Inference Engineering · Day 8 · Evening

A process ID is not enough to prove process identity.

Windows can recycle a PID after a model worker exits. If cleanup logic trusts the number alone, it may block on an unrelated process—or terminate the wrong one.

Lizard records PID plus start time and verifies the descendant tree before declaring model memory released. A crashed worker becomes recoverable without making cleanup reckless.

Inference reliability includes the process lifecycle around the kernel.

Engineering fact: Lizard tracks process identity using PID plus start time so a recycled Windows PID is not mistaken for the old model process.

Read the relevant Lizard page

#WindowsEngineering #ReliabilityEngineering #ProcessManagement #LocalAI #LizardLLM

<!-- lizard-marketing-slot:day-08-pm -->

Inference Engineering · Day 8 · Morning

A missing baseline should produce a failed preflight—not a blank treated as zero.

In this run, the requested Ollama model was not installed. Lizard recorded the exact command, failure reason and exit status, then refused to manufacture an Ollama measurement.

The later Native, Caterpillar and llama.cpp run is valid for those providers. Ollama remains unmeasured.

Reliable benchmarking begins by preserving failure evidence.

Engineering fact: When the requested Ollama model is absent, Lizard records the failed preflight and produces no synthetic Ollama measurement.

Read the relevant Lizard page

#Benchmarking #Reproducibility #LocalAI #LLMInference #LizardLLM

<!-- lizard-marketing-slot:day-08-am -->

Inference Engineering · Day 7 · Evening

We implemented an AVX-512 VNNI decode kernel and chose not to enable it by default.

The instruction is promising on supported Intel and AMD CPUs. But on the hardware available for validation, the measured result remained inside ordinary run-to-run noise.

Caterpillar ships the proven path as default and keeps VNNI behind an explicit experiment flag.

Engineering credibility sometimes means declining to claim a speedup.

Engineering fact: Caterpillar's AVX-512 VNNI kernel remains opt-in because it measured within normal run-to-run noise on available test hardware.

Read the relevant Lizard page

#AVX512 #CPUOptimization #Benchmarking #CaterpillarEngine #LizardLLM

<!-- lizard-marketing-slot:day-07-pm -->

Inference Engineering · Day 7 · Morning

KV-cache is one of those pieces that only looks invisible until a long chat starts crowding out memory. Each generated token adds keys and values that have to stay around for later attention, so cache precision is not a trivia setting. It is part of the capacity plan.

lizard-native makes that choice explicit with f16, q8_0, and q4_0 KV-cache precision. That matters because the trade is real: preserve more fidelity, or fit more context inside the same machine budget. When the decision is hidden, people tend to blame the model when the actual limit is memory.

The useful part is not that one setting is always best. It is that the runtime surfaces the cost where the user can reason about it. On a laptop or a small workstation, that changes the conversation from “why did this stop?” to “what do I want to spend memory on?”

In practice, I would rather see a clear cache trade than a vague “supported” label. It gives operators something they can tune against the session length they actually need, instead of discovering the limit after the context has already grown. When you are sizing local inference for real users, do you bias toward higher cache precision or more room for conversation length?

Engineering fact: lizard-native exposes f16, q8_0, and q4_0 KV-cache precision as an explicit context-length versus memory trade.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#KVCache #ContextWindow #LocalInference #SystemsEngineering #WindowsAI

<!-- lizard-marketing-slot:day-07-am -->

Inference Engineering · Day 6 · Evening

An integrated GPU is not a small discrete GPU.

On unified-memory hardware, CPU and GPU already access the same physical memory. Treating that system like a discrete card can add unnecessary staging and copies.

lizard-native includes a UMA zero-copy path for supported integrated graphics. The design goal is simple: shared weights should stay shared, while the Direct3D 12 execution path remains explicit.

Hardware-aware inference starts by respecting the memory architecture you actually have.

Engineering fact: lizard-native includes a zero-copy path for unified-memory integrated GPUs so shared memory is treated as shared rather than copied as if discrete.

Read the relevant Lizard page

#LizardNative #IntegratedGPU #D3D12 #ZeroCopy #LocalAI

<!-- lizard-marketing-slot:day-06-pm -->

Inference Engineering · Day 6 · Morning

A decode loop can look GPU-bound and still waste time on the host.

In the lizard-native Gemma path, roughly 400 weight descriptors were being rebuilt for every generated token. That meant the runtime kept reconstructing the same per-token metadata before the next step could move forward. The GPU had work to do, but so did the scaffolding around it, and that scaffolding sat on the critical path.

The fix was to stop rebuilding those descriptors and keep them around for reuse. On the Gemma test runs, that improved decode by 13–26% and brought first-token response back by about 20%.

The useful lesson is that decode tuning is rarely only about the kernel. If the loop repeats cleanly, inspect the state that gets recreated every pass: descriptor setup, graph metadata, plan assembly, session bookkeeping, cache wiring. Those are easy to ignore because they do not look like compute, but they can still decide how fast the next token appears.

I have learned to ask a boring question before touching the math: what is being rebuilt on every token that could have been carried forward from the previous one?

In practice, persistent per-token state often buys a cleaner win than shaving a few cycles off the math itself. Less allocation, less synchronization pressure, fewer host-side stalls, and a shorter path back into the decode loop. When the hot path is stable, that kind of cleanup tends to show up immediately in the first-token feel as well as steady-state throughput.

Engineering fact: The lizard-native Gemma path stopped rebuilding roughly 400 weight descriptors per token; test runs improved decode 13–26% and first-token response about 20%.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#localai #inferencetuning #decodeperformance #windowsai #d3d12

<!-- lizard-marketing-slot:day-06-am -->

Caterpillar benchmark CLI and HTTP guide

The provider flag is explicit and verified. The health response and artifact
must both report runtimeProvider: caterpillar; the harness fails on a
provider mismatch.

npm run bench:native:http -- `
  --provider caterpillar `
  --model llama-3.2-3b-instruct `
  --gguf "$env:USERPROFILE\.lizard-llm\models\llama-3.2-3b-instruct.gguf" `
  --batch "16,8,4,1" `
  --tokens 64 `
  --label caterpillar-llama32-same-weights-http

Use --skip-llama for Caterpillar-only tuning. Verify the runtime provider
before interpreting any number.

Adaptive concurrency

The native policy uses B=16 as the aggregate-throughput target and B=8 as
the normal secondary lane. B=4 and B=1 are used only when both primary lanes
are unavailable. Automatic result selection chooses the faster successful
B=16/B=8 measurement; a failed or token-unmeasured lane is never promoted.

What a valid llama.cpp ratio requires

The candidate and control must load the same physical GGUF file. Keep the
model, quantization, prompt, output token budget, concurrency, warm-up and
machine conditions fixed. A Llama 3.2 candidate cannot be compared with a Gemma
control. The HTTP harness refuses a mismatched canonical path.

The artifact records provider identity, candidate/control paths, completion and
failure counts, aggregate completion tokens per second, full-response latency,
lane selection and provenance. The current non-streaming harness does not claim
TTFT or TPOT.

Read the complete command reference

The public guide includes copy buttons, parameters, output interpretation,
throughput and latency runs, serving, offline and online checks, repeated-prefix
workloads, sequence-length and stress runs, production checks, and current
multi-GPU/tensor-parallel/data-parallel limitations:

https://lizard-llm.qendryx.com/benchmark-cli.html

Benchmark methodology and screenshots: https://lizard-llm.qendryx.com/benchmarks.html
Technical architecture: https://lizard-llm.qendryx.com/technical-overview.html

Lizard Native benchmark CLI and HTTP guide

Run the user-visible OpenAI-compatible serve path against a stock llama.cpp
server with the same Llama 3.2 GGUF:

npm run bench:native:http -- `
  --provider lizard-native `
  --model llama-3.2-3b-instruct `
  --gguf "$env:USERPROFILE\.lizard-llm\models\llama-3.2-3b-instruct.gguf" `
  --batch "16,8,4,1" `
  --tokens 64 `
  --label lizard-native-llama32-same-weights-http

Use --skip-llama for provider-only tuning. That mode produces no admissible
cross-runtime ratio.

Adaptive concurrency

The native policy uses B=16 as the aggregate-throughput target and B=8 as
the normal secondary lane. B=4 and B=1 are used only when both primary lanes
are unavailable. Automatic result selection chooses the faster successful
B=16/B=8 measurement; a failed or token-unmeasured lane is never promoted.

What a valid llama.cpp ratio requires

The candidate and control must load the same physical GGUF file. Keep the
model, quantization, prompt, output token budget, concurrency, warm-up and
machine conditions fixed. A Llama 3.2 candidate cannot be compared with a Gemma
control. The HTTP harness refuses a mismatched canonical path.

The artifact records provider identity, candidate/control paths, completion and
failure counts, aggregate completion tokens per second, full-response latency,
lane selection and provenance. The current non-streaming harness does not claim
TTFT or TPOT.

Read the complete command reference

The public guide includes copy buttons, parameters, output interpretation,
throughput and latency runs, serving, offline and online checks, repeated-prefix
workloads, sequence-length and stress runs, production checks, and current
multi-GPU/tensor-parallel/data-parallel limitations:

https://lizard-llm.qendryx.com/benchmark-cli.html

Benchmark methodology and screenshots: https://lizard-llm.qendryx.com/benchmarks.html
Technical architecture: https://lizard-llm.qendryx.com/technical-overview.html

Lizard wants you

We are looking for 100 people to test the new Lizard Native AI inference
provider on hardware we do not own. Free access, no cost and no catch.

What you get

• Free access. Both native engines, unlocked, for taking part. There is no card to enter and nothing is charged.
• Benchmarks on your own hardware. The measurements come from your machine, not ours — we cannot buy every configuration our users own.
• A say in what gets built. What testers report in public is what gets fixed first.

What we ask in return

One thing, and this is the important part:

1. Install Lizard and run the built-in benchmark on your own machine. It is one click.
2. Post the result here, in this community, with your hardware next to it — processor, memory, graphics card, and which engine ran. A tokens-per-second figure with no machine attached tells nobody anything.
3. Tell us what should improve. Slow, confusing, broken or missing — say so here, in public. A complaint with a benchmark attached is the most useful thing you can send us.

Benchmarks and improvement reports both belong in the forum, not in a private
inbox. That is the whole point: the numbers are only worth something if other
people can read them and compare their own.

Who we are looking for

• 64-bit Windows 10 or 11 — the odder the hardware the better
• Integrated graphics, older GPUs and laptops especially: that is where local AI is hardest and least measured
• Willing to post publicly under your own name or handle
• No AI or programming background required

---

Apply here — three fields, no account needed: https://lizard-llm.qendryx.com/testers.html

Inference Engineering · Day 5 · Evening

What if one pass over the model weights could accept several tokens?

Caterpillar's speculative path drafts a short sequence, then verifies multiple candidates in a single weight pass. It is lossless: rejected drafts fall back to the normal path, so accepted output remains the output the model would have produced.

On predictable or repetitive text, our tests reached up to 2× speed. On ordinary prose, the path steps aside rather than forcing a bad optimization.

Optimization should be conditional when the workload is conditional.

Engineering fact: Caterpillar's lossless speculative path can verify multiple drafted tokens per weight pass on predictable output and steps aside on ordinary prose.

Read the relevant Lizard page

#SpeculativeDecoding #CaterpillarEngine #LLMInference #LocalAI #LizardLLM

<!-- lizard-marketing-slot:day-05-pm -->

Inference Engineering · Day 5 · Morning

Caterpillar now defaults to physical cores, and that change came from a simple result: on a four-core test laptop, using every logical thread made decode slower. Switching to the physical-core count improved measured decode by about 9% and also made the timing less noisy.

The important part is not the exact laptop. It is the shape of the workload. Decode can be limited by memory bandwidth long before it is limited by raw thread count. Once that happens, hyperthreads stop looking like extra capacity and start looking like contention. More schedulable threads can mean more pressure on the same caches, the same memory channels, and the same execution resources.

That is why the default changed. A local runtime should choose the thread count that matches the bottleneck, not the number that looks best on a spec sheet. The override still matters for experiments and unusual machines, but the default should bias toward the setting that is most likely to behave well without tuning.

I have seen this pattern enough times to treat thread count as part of measurement, not just configuration. If a workload is bandwidth-bound, the “more threads” instinct can be the wrong first move.

What’s the first sign you use to tell whether extra logical threads are helping, or just adding noise?

Engineering fact: Caterpillar defaults to physical cores; on a four-core test laptop this improved decode by about 9% versus using every logical thread.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#localai #cpu #performance #benchmarking #systems

<!-- lizard-marketing-slot:day-05-am -->

Inference Engineering · Day 4 · Evening

The Q4 change was a layout rewrite, not a new decode path.

Caterpillar's four-bit weights were repacked around eight-row tiles, and the scale data now gets unpacked once at load instead of being pulled back through the hot loop on every token. That sounds like a small structural change, but on test hardware it cut bytes read per token by about 13% and improved decode by about 8% while keeping the output byte-identical.

The practical lesson is simple: once decode is memory-bound, the difference between “prepare it once” and “touch it every token” starts to show up as throughput. The math did not change. The answer did not change. The memory traffic did.

That is the part worth watching in your own kernels and data paths. If a format change makes the hot loop carry less baggage, you often get the speedup without any behavioral risk.

Have you seen a layout change beat a kernel rewrite on a workload you care about?

Engineering fact: On Lizard's test hardware, Caterpillar's eight-row Q4 layout reduced bytes read per token by about 13% and improved decode by about 8% with byte-identical output.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#Caterpillar #Q4 #LLMInference #SystemsProgramming #PerformanceEngineering

<!-- lizard-marketing-slot:day-04-pm -->

Inference Engineering · Day 4 · Morning

Peak FLOPS is a poor shorthand for decode performance on local LLMs.

Autoregressive generation reads the model weights again for every token. That means the cost you feel first is often bytes moved, not raw arithmetic capacity. On consumer hardware, memory bandwidth can become the ceiling before the compute units are busy enough to matter.

That shifts the tuning order. If decode is the bottleneck, the practical questions are about weight traffic, quantization, layout, and cache behavior. A smaller quantization can help twice: fewer bytes to move and less pressure on the memory subsystem. In that case, adding threads or chasing a bigger peak FLOPS number may change less than expected.

For a local runtime, this is the difference between a model that looks large on paper and one that actually keeps up during generation. It also explains why decode profiling needs a bytes-per-token view, not just a “compute utilization” view.

When you profile local decode, what tends to move the needle more for you: quantization choice, KV-cache behavior, or memory layout?

Engineering fact: Autoregressive decode repeatedly reads model weights, so bytes moved per token often matter more than peak arithmetic throughput.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#llm #decode #memorybandwidth #quantization #localai

<!-- lizard-marketing-slot:day-04-am -->

Inference Engineering · Day 3 · Evening

If you trace token latency in a local runtime, the obvious place to look is often the wrong one.

The model kernels can look healthy while the loop still loses time to coordination: CPU work sets up the next step, the GPU waits for a handoff, and the runtime repeats the same planning work for every token. That kind of overhead is small in isolation, but over a long decode it becomes part of the profile you actually feel.

Caterpillar changes that by rebuilding the execution core around a pre-compiled compute-graph plan. The useful effect is not just that the graph exists. It is that the runtime can stop reconstructing the same execution shape on every token and instead follow a plan that is already laid out. That reduces per-token CPUGPU synchronization, which is exactly the sort of cost that hides between kernel launches.

The practical lesson is to separate math cost from orchestration cost. When I look at a slow generation path, I want to know whether the runtime is paying for repeated scheduling, repeated graph setup, or repeated handoff boundaries that could have been planned once. If the execution shape is stable, the hot path gets simpler. If it is not, even good kernels can end up carrying latency that never shows up in FLOPS.

One thing I have found useful is to inspect the decode path at the token boundary, not just the kernel level. If the coordination layer is doing more work than expected, the fix is usually structural rather than numerical.

Have you seen a case where reducing CPUGPU synchronization moved token latency more than changing the kernel itself?

Engineering fact: Caterpillar rebuilds the execution core around a pre-compiled compute-graph plan to reduce per-token CPUGPU synchronization.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#Caterpillar #ComputeGraph #Inference #WindowsAI #LocalLLM

<!-- lizard-marketing-slot:day-03-pm -->

Inference Engineering · Day 3 · Morning

A startup trace is easier to trust when one expensive step stops repeating.

Shader compilation is a good example. If the first launch is slow, that is expected on some GPU paths. The more useful question is whether later launches are still paying the same warm-up cost. When that happens, the slowdown is not really about inference speed. It is repeated setup work hiding in the launch path.

Both native engines now persist compiled GPU shaders, so the first warm-up is paid once instead of on every load. That changes how I read the trace. A cold start can still be expensive, but a warm start should converge. If it does not, I want to look at cache persistence, residency decisions, and any path that forces the runtime to rebuild GPU work before it reaches the model.

In practice I split startup into three buckets: cold load, warm load, and steady-state decode. Cold load tells me what the system pays to become ready. Warm load tells me whether setup work is being repeated. Steady-state decode is where model-path changes should show up clearly. If those three are mixed together, you end up tuning the wrong layer and the numbers stay noisy.

That is why this small persistence change matters. It makes startup behavior more legible. A trace that used to look like a slow model can now point to the real culprit: repeated shader compilation, repeated cache rebuilds, or a launch sequence that is doing more work than it should.

When you profile GPU startup, what is the first thing you separate: cold load, warm load, or decode?

Engineering fact: Both native engines persist compiled GPU shaders so the first warm-up is paid once instead of on every load.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#GPUShaders #ShaderCompilation #StartupLatency #NativeRuntime #PerformanceDebugging

<!-- lizard-marketing-slot:day-03-am -->

Inference Engineering · Day 2 · Evening

A model that almost fits can create the worst failure mode: it runs, but it thrashes.

On a tight GPU budget, repeatedly evicting and re-uploading layers can reduce decode to a crawl. A single “GPU or CPU” decision is too coarse.

Lizard plans prefill and decode independently. It can keep GPU-assisted prompt processing while moving decode to a bounded, stable path when full residency is unsafe.

The engineering lesson: split the phases before you split the blame.

Engineering fact: When a model exceeds the usable GPU weight budget, Lizard can keep GPU-assisted prompt processing and use bounded hybrid decode instead of repeated eviction.

Read the relevant Lizard page

#VRAM #LLMInference #GPUPerformance #LizardNative #LocalAI

<!-- lizard-marketing-slot:day-02-pm -->