Architecture

Lizard inference engineering: Run the real HTTP readiness check

lizardadmin — Sat, 22 Aug 2026 11:00:14 GMT

Inference Engineering · Day 30 · Evening [image: Screenshot%202026-07-22%20221220.png] If you want a readiness check that means something, run the HTTP benchmark against a real model file instead of a toy case. The useful part here is not the peak number on a single pass. It is the comparison across the same GGUF with two batch sizes, B=16 and B=8, and the ability to line that up against llama.cpp on the same host. That matters because batch size changes the shape of the request. A model can look healthy at one setting and expose a different bottleneck at the other. Memory pressure, launch overhead, and decode behavior do not always move together. When you keep the artifact constant and only change the runtime lane, you get a cleaner view of what the engine is actually doing. The practical habit is simple: capture the command, keep the GGUF path fixed, save the benchmark artifact, and record the hardware context along with the result. That gives you something a second person can rerun later without guessing which variable changed. I have found that the most useful benchmark writeup answers three things: did the model load, did the request complete, and where did the two runtimes diverge under the same input. If you only keep one number, you lose the signal. What model and Windows machine would you use as the first real HTTP readiness check? hashtags":["LizardLLM","HTTPBenchmark","llamaCpp","GGUF","InferenceEngineering"]}, Engineering fact: The Lizard Console command lizard-llm benchmark http can test lizard-native or Caterpillar at B=16 and B=8 and compare the same GGUF with llama.cpp. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #LizardLLM #HTTPBenchmark #llamaCpp #GGUF #InferenceEngineering

Lizard inference engineering: Community questions should keep users anonymous

lizardadmin — Fri, 21 Aug 2026 11:00:11 GMT

Inference Engineering · Day 29 · Evening [image: Screenshot%202026-07-22%20213650.png] Community incident workflows work better when they separate the technical problem from the person who hit it. If a major failure is shared to the Lizard Community, the workflow publishes an anonymized topic and keeps personal user information out of the public discussion. That matters more than it sounds. Once a thread carries a name, account detail, or other identifying context, people start answering the person instead of the failure mode. The result is usually less useful debugging and more friction around what should have been a clean technical exchange. The practical pattern is simple: store the private incident record where it belongs, then open the public discussion with only the facts that help others reproduce, recognize, or explain the issue. Keep the Ticket ID linkage behind the curtain. Keep the public thread focused on symptoms, environment, model or runtime behavior, and the steps already ruled out. That gives engineers enough surface area to reason about the fault while preserving the user’s privacy boundary. I like this design because it lowers the cost of asking for help. People are more willing to report a failure when they know the community thread will not expose them by default. It also keeps the public archive cleaner for future readers, since anonymized incident titles are easier to search, group, and reuse than threads built around a single user’s account history. What diagnostic details have you found are safe to share publicly without turning the thread into a privacy risk? Engineering fact: When a major failure is shared to the Lizard Community, the incident workflow publishes an anonymized topic and keeps personal user information out of the public discussion. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #privacy #incidentresponse #communitysupport #debugging #softwareengineering

Lizard inference engineering: Keep user metrics attached to the response

lizardadmin — Thu, 20 Aug 2026 01:00:08 GMT

Inference Engineering · Day 28 · Morning [image: Screenshot%202026-07-22%20213650.png] A separate dashboard is useful, but the response already knows how it was served. Lizard's native HTTP response metadata can carry the current user-visible measurement, the comparable history summary, runtime provider, and selected concurrency policy. The client can inspect the result without guessing which backend handled it. Observability is strongest when it travels with the event it describes. Which serving details would your client application surface beside a local AI response? Engineering fact: Lizard's OpenAI-compatible native response metadata can include the current HTTP measurement, historical user summary, provider identity, and concurrency policy. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Onboarding should recommend only compatible models

lizardadmin — Mon, 17 Aug 2026 01:00:11 GMT

Inference Engineering · Day 25 · Morning [image: Screenshot%202026-07-22%20213650.png] Onboarding gets more useful when it stops treating every model as a possible choice. The practical step is simple: check the model metadata, the quantization target, the machine’s detected hardware, and the estimated memory footprint before the user sees a recommendation. Lizard does that up front, then presents a native choice only when the model fits the current system. That matters because a bad recommendation is not a minor UX flaw. It sends people into dead ends: wrong quantization for the available backend, a model that overshoots memory, or a file that looks plausible but will never run well on the detected hardware. If the flow knows the machine cannot carry the load, it should say so plainly and keep the rest of the list out of the way. The useful part is not just rejection. It is a recommendation with context: why this model fits, what constraint made it pass, and what changed if the user switches hardware or memory budget. That gives the onboarding flow something closer to an engineering check than a browse-and-hope catalog. In practice, this also keeps the local model path honest. A recommendation should reflect what the runtime can actually execute, not what a registry happens to offer. What one signal do you trust most when you decide whether a local model is safe to recommend before download? Engineering fact: Lizard combines model metadata, quantization compatibility, hardware capability, and memory estimates before presenting a native recommendation. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #modelonboarding #modelselection #windowsai #llmops

Lizard inference engineering: A restart should remember the last valid runtime

lizardadmin — Sun, 16 Aug 2026 11:00:08 GMT

Inference Engineering · Day 24 · Evening [image: Screenshot%202026-07-22%20213650.png] Freeing memory and forgetting configuration are separate actions. Lizard records the last stopped model, runtime provider, and relevant profile state after release is proven. That gives the operator a defined restart path without claiming the model remained resident. Lifecycle state becomes easier to reason about when 'stopped', 'restartable', and 'loaded' are distinct. What state should survive when you unload a local model? Engineering fact: Lizard's model lifecycle retains restartable metadata for the last stopped model and provider so an intentional stop can be reversed without rediscovering the configuration from scratch. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Model release must be proven, not assumed

lizardadmin — Sun, 16 Aug 2026 01:00:11 GMT

Inference Engineering · Day 24 · Morning [image: Screenshot%202026-07-22%20213650.png] A stop command returning is not the same thing as model memory being released. The failure mode is easy to miss in a local runtime: the command reports success, the UI moves on, and the next load is expected to behave as if the old model is gone. But if a child process still owns mappings, handles, or GPU allocations, that assumption is wrong. You can end up reading stale available-memory numbers, reusing a slot too early, or debugging a “ghost model” that was never actually unloaded. The useful discipline here is to tie release to process identity, not to command status. Lizard tracks the model-owning process tree and records release proof only when the owning processes actually exit. That gives you something concrete to inspect later: not just that a stop was requested, but that the process tree which held residency is gone. That matters when you are deciding whether to load a larger model, whether to trust a memory reading, or whether a failed unload needs to be retried instead of ignored. It also makes the lifecycle easier to reason about in mixed setups where one engine spawns helpers and another one keeps a worker warm. If the owning tree is still alive, residency is still real. I’ve found this is one of those details that only seems optional until the first time a large load collides with a process that was assumed dead. What’s the most reliable signal your stack uses to prove the previous model actually left memory? Engineering fact: Lizard tracks the model-owning process tree and records release proof so a stopped model is not declared unloaded merely because a command returned. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #modellifecycle #processtracking #localai #windowsai #gpu

Lizard inference engineering: Unknown quantization should stay unknown

lizardadmin — Sat, 15 Aug 2026 11:00:06 GMT

Inference Engineering · Day 23 · Evening [image: Screenshot%202026-07-22%20213650.png] Unknown quantization is worth treating as a boundary, not a guess. If a GGUF file carries quantization metadata that Lizard does not recognize for the selected model family, the runtime does not promote it into a supported default. It reports the compatibility reason directly, which keeps the plan honest and gives the rest of the flow something concrete to work with. That matters in a local stack because model files and quantization variants move faster than runtime support. A silent fallback can hide the real problem for too long: maybe the file is fine, maybe the family match is wrong, or maybe the format simply needs a newer engine path. Those are very different operational outcomes, and they should not collapse into one vague “best effort” label. In practice, this makes the recommendation path easier to trust. A user can see why a model was rejected, compare it with a compatible alternative, and avoid burning time on a format that only looks close enough. It also keeps logs and support cases readable, which is usually where these edge cases get expensive. I have found that the most useful compatibility message is the one that names the boundary plainly and stops there. No invented capability, no implied default, no optimism tax. When you are maintaining a local model runtime, do you prefer the tool to reject unknown formats immediately, or to try a heuristic path first and explain afterward? Engineering fact: When Lizard encounters an unknown or unsupported native quantization, it reports the compatibility reason rather than treating the format as a supported default. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #quantization #gguf #localai #modelruntime #llmops

Lizard inference engineering: Quantization changes more than file size

lizardadmin — Sat, 15 Aug 2026 01:00:08 GMT

Inference Engineering · Day 23 · Morning [image: Screenshot%202026-07-22%20213650.png] Quantization changes more than file size. It changes the memory budget, the kernel path, and whether a model family can run natively at all. That is why compatibility has to treat quantization as an execution decision, not a cosmetic one. A model in F32 or F16 is not asking the same thing from the runtime as Q8_0, Q6_K, Q5_K_M, Q4_K_M, or IQ4_XS. The supported path depends on the family as well as the tensor type, so the check has to happen before execution starts. The practical effect shows up fast on limited hardware. A smaller file can still push a machine over the edge if the dequant work or memory layout is wrong for the target backend. The reverse is also true: a heavier format may be the safer choice when you need predictable quality and the model family supports it cleanly. In day-to-day planning, I find it helps to ask three questions in order: will it fit, will it execute on the native path, and what does the decode path cost once the model is loaded? That keeps the conversation grounded in what the machine can actually do instead of what the filename suggests. Which quantization choice do you end up optimizing for most often on real hardware: fit, quality, or decode speed? Engineering fact: Lizard's compatibility layer treats quantization as an execution capability and memory decision; supported native formats include F32, F16, BF16, Q8_0, Q6_K, Q5_K_M, Q4_K_M, and IQ4_XS with family-specific constraints. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #quantization #llm #localai #inference #windows

Lizard inference engineering: Keep local serving observable

lizardadmin — Wed, 12 Aug 2026 11:00:09 GMT

Inference Engineering · Day 20 · Evening [image: Screenshot%202026-07-22%20213650.png] Local does not have to mean opaque. Lizard's native server surfaces the provider and model it actually loaded, the concurrency policy it selected, and any matching measured HTTP profile. Response metadata also carries serving-path and user-performance information. You should be able to explain a slow local reply without attaching a debugger first. What would you want on the first screen of a local inference health check? Engineering fact: The native HTTP health and metrics endpoints expose runtime provider, model identity, selected concurrency policy, and measured HTTP performance evidence. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Use a familiar local HTTP contract

lizardadmin — Wed, 12 Aug 2026 01:00:09 GMT

Inference Engineering · Day 20 · Morning [image: Screenshot%202026-07-22%20213650.png] A new runtime should not require every client application to be rewritten. Lizard exposes a local OpenAI-compatible chat-completions endpoint for supported models. Existing clients can point at localhost while Lizard owns model loading, provider selection, concurrency, and native telemetry behind the contract. Compatibility at the API boundary keeps runtime experimentation from leaking into every application. Which local tool would you connect first if the endpoint already matched your current client? Engineering fact: Lizard can serve supported local models through an OpenAI-compatible chat-completions endpoint on localhost for integration with existing clients. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: One Ticket ID should connect the whole incident

lizardadmin — Sun, 09 Aug 2026 11:00:07 GMT

Inference Engineering · Day 17 · Evening [image: Screenshot%202026-07-22%20213650.png] A ticket number is only useful if every part of the incident points back to it. For a major failure, Lizard stores diagnostics under the Ticket ID and tracks notification delivery plus the anonymized community discussion URL in the same incident record. Repeated occurrences reuse the trace instead of fragmenting it. That gives the user and engineering team one durable reference from failure through resolution. Can your users follow a critical runtime failure after the first error dialog closes? Engineering fact: A major Lizard incident links its Ticket ID, local diagnostics, delivery state, engineering notification, and anonymized community discussion in one lifecycle record. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Critical failures deserve a ticket, warnings do not

lizardadmin — Sun, 09 Aug 2026 01:00:08 GMT

Inference Engineering · Day 17 · Morning [image: Screenshot%202026-07-22%20213650.png] Automatic failure reporting becomes noise if every warning opens an incident. Lizard's Major Failure Intelligence path is reserved for critical conditions: failed model initialization, corrupted GGUFs, unsupported hardware, repeated crashes, and unrecoverable runtime errors. Minor warnings remain local diagnostics. The threshold protects both user trust and engineering attention. Which failure classes in your product deserve automatic escalation? Engineering fact: Lizard's Major Failure Intelligence classifies incidents and creates a Ticket ID only for critical conditions such as failed initialization, corrupted GGUFs, unsupported hardware, or unrecoverable runtime failures. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Unsupported models should fail before inference

lizardadmin — Sat, 08 Aug 2026 01:00:10 GMT

Inference Engineering · Day 16 · Morning [image: Screenshot%202026-07-22%20213650.png] The safest failure happens before the model starts generating. Lizard evaluates GGUF architecture and quantization support before selecting the native executor. Unsupported combinations can be explained and routed through the compatibility and recommendation flow instead of producing a mysterious partial run. Compatibility is a user-facing decision, not just an internal exception. What does your onboarding flow show when a newly released GGUF family is not ready yet? Engineering fact: Lizard inspects GGUF architecture and quantization compatibility before native execution and can route unsupported combinations to compatible alternatives instead of attempting an unsafe run. Read the relevant Lizard page #LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering

Lizard inference engineering: Exact token IDs matter at the prompt boundary

lizardadmin — Thu, 06 Aug 2026 03:13:06 GMT

Inference Engineering · Day 16 · Evening [image: Screenshot%202026-07-22%20213650.png] A token estimate is fine for planning memory, but it is the wrong artifact to hand to native prefill. At the prompt boundary, the model needs the exact token sequence it was trained to consume. If the GGUF tokenizer metadata cannot produce those token IDs, Lizard refuses native prefill instead of substituting a guessed token count and hoping the execution path stays correct. That matters because a bad estimate is not a small arithmetic error once the prompt is on the wire. It can shift cache allocation, change the sequence handed to the model, and make the runtime look healthy while it is actually operating on the wrong input. Refusing the request is less convenient, but it keeps execution aligned with the real prompt rather than an approximation of it. In practice, this is the kind of boundary worth making explicit in any inference system: estimation is for planning, exact tokens are for execution. If the tokenizer metadata cannot support exact IDs, the safe answer is to stop. Where do you draw that line in your own inference stack? Engineering fact: Lizard refuses native prefill when it cannot produce exact prompt token IDs from the GGUF tokenizer metadata rather than substituting an estimated token count. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #gguf #tokenization #inference #promptengineering #llmops

Lizard inference engineering: Keep the API local and familiar

lizardadmin — Sun, 02 Aug 2026 01:00:11 GMT

Inference Engineering · Day 10 · Morning [image: Screenshot%202026-07-22%20213650.png] If you are trying to keep a local runtime easy to adopt, the API shape matters as much as the model choice. After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint. That means an existing app can usually keep its client logic, swap the base URL, and keep moving. You still get local prompts and outputs, which is the part that usually decides whether a team treats a runtime as a tool or as a project. That compatibility also changes how you test integrations. You can validate request format, retry behavior, and response handling against a local endpoint before you ever point anything at a remote provider. For Windows teams especially, that is a practical way to reduce friction without giving up the familiar /v1/chat/completions style contract. The lesson is simple: local-first works better when it changes the data path, not the application’s expectations. If the endpoint stays familiar, the migration is mostly about deployment and model choice instead of rewriting every client. When you’ve brought a local model endpoint into an existing app, what usually breaks first: auth assumptions, streaming, or tool-calling shape? Engineering fact: After activation, Lizard serves chat and generation through a local OpenAI-compatible endpoint while prompts and outputs stay on the machine. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #localai #openaicompatible #windows #llmops #apidesign

Lizard inference engineering: Prove a process really exited

lizardadmin — Fri, 31 Jul 2026 11:00:11 GMT

Inference Engineering · Day 8 · Evening [image: Screenshot%202026-07-22%20214040.png] A process ID is not enough to prove process identity. Windows can recycle a PID after a model worker exits. If cleanup logic trusts the number alone, it may block on an unrelated process—or terminate the wrong one. Lizard records PID plus start time and verifies the descendant tree before declaring model memory released. A crashed worker becomes recoverable without making cleanup reckless. Inference reliability includes the process lifecycle around the kernel. Engineering fact: Lizard tracks process identity using PID plus start time so a recycled Windows PID is not mistaken for the old model process. Read the relevant Lizard page #WindowsEngineering #ReliabilityEngineering #ProcessManagement #LocalAI #LizardLLM

Lizard inference engineering: Decode is a memory-bandwidth problem

lizardadmin — Mon, 27 Jul 2026 01:00:11 GMT

Inference Engineering · Day 4 · Morning [image: day-04-am-universal-1785117617625.png] Peak FLOPS is a poor shorthand for decode performance on local LLMs. Autoregressive generation reads the model weights again for every token. That means the cost you feel first is often bytes moved, not raw arithmetic capacity. On consumer hardware, memory bandwidth can become the ceiling before the compute units are busy enough to matter. That shifts the tuning order. If decode is the bottleneck, the practical questions are about weight traffic, quantization, layout, and cache behavior. A smaller quantization can help twice: fewer bytes to move and less pressure on the memory subsystem. In that case, adding threads or chasing a bigger peak FLOPS number may change less than expected. For a local runtime, this is the difference between a model that looks large on paper and one that actually keeps up during generation. It also explains why decode profiling needs a bytes-per-token view, not just a “compute utilization” view. When you profile local decode, what tends to move the needle more for you: quantization choice, KV-cache behavior, or memory layout? Engineering fact: Autoregressive decode repeatedly reads model weights, so bytes moved per token often matter more than peak arithmetic throughput. Lizard The AI Runtime You'll Own—Not Rent. Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency. Read the relevant Lizard page #llm #decode #memorybandwidth #quantization #localai

Lizard inference engineering: An inference provider owns the stack

lizardadmin — Fri, 24 Jul 2026 01:00:10 GMT

Inference Engineering · Day 1 · Morning An API endpoint is not an inference engine. The engineering decisions that determine latency happen below the route: batching, KV-cache precision, quantized kernels, memory residency and device submission. Lizard treats the local machine as the inference provider. lizard-native owns the Direct3D 12 path; Caterpillar owns its compiled execution plan. The OpenAI-compatible endpoint is only the top layer. Which layer of your current inference stack can you actually inspect and change? Engineering fact: Lizard owns the local endpoint, scheduler, cache layout, kernels, quantization choices, and device execution path. Read the relevant Lizard page #InferenceEngineering #LocalAI #OnDeviceAI #LizardLLM #LLMEngineering

Technical overview: Lizard Native and Caterpillar vs llama.cpp and Ollama

lizardadmin — Thu, 23 Jul 2026 05:44:14 GMT

The new architecture overview explains why these four names are not interchangeable categories. [image: technical-overview-dual-engine.webp] What each layer is for Lizard Native — a specialized Windows-first resident-GPU provider using Direct3D 12 for its supported native subset. Caterpillar — a standalone clean-room provider built around a typed DAG, reused activation arena, and compiled execution plan. llama.cpp — a broad cross-platform inference toolkit and server with many CPU and GPU backends, wide model coverage, continuous batching, embeddings, reranking, and multimodal features. Ollama — model packaging, acquisition, lifecycle, scheduling, and local APIs, including configurable model keep-alive and OpenAI-compatible endpoints. Lizard's extra control-plane work is the hardware scan, exact GGUF fit decision, native-provider routing, and provider-separated evidence. llama.cpp and Ollama remain useful stock baselines and compatibility fallbacks. No provider wins every model, metric, or machine. Primary baseline documentation: llama.cpp project llama-server Ollama runtime FAQ Ollama OpenAI compatibility Read the complete comparison table: https://lizard-llm.qendryx.com/technical-overview.html Question: Which comparison dimension matters most to you: platform coverage, model lifecycle, memory fit, or provider-level benchmark evidence?

Two engines, one decision: how Lizard routes your model

lizardadmin — Sun, 19 Jul 2026 11:26:00 GMT

Lizard ships two execution engines and picks between them for you. This is the part people ask about most, so here is the actual decision. [image: two-engines-decision.png] The question it asks is simple: does this model fit in your GPU's memory, alongside the KV cache and everything else already on the card? If yes → lizard-native. Weights go resident in VRAM, work runs through Direct3D 12, and this is the fastest path available on your hardware. If no, or there is no discrete GPU → caterpillar. This is the newer engine, built around a pre-compiled execution plan rather than interpreting a graph each token. It is designed for the case most benchmarks ignore: a normal laptop with integrated graphics. A few things worth knowing about caterpillar specifically, because they are unusual: The weight layout is the optimisation. Decoding is memory-bound, so the win came from reading fewer bytes per token, not from doing arithmetic faster. Restructuring the four-bit weight layout into eight-row tiles cut bytes-read per token by about 13% and decode time by about 8% — with byte-identical output. Speculative decoding is lossless. On predictable text it drafts several tokens and verifies them in a single pass over the weights, then keeps only the ones the real model agrees with. Same output, fewer passes. On ordinary prose the acceptance rate drops, so it stops trying rather than making you pay for failed guesses. Physical cores beat logical threads. Because the limit is memory bandwidth, using every hyperthread adds contention without adding bandwidth. Four threads measured faster than eight on a 4-core laptop. An honest note on where this sits. Measured against stock llama.cpp on the same machine, caterpillar's general-case decode is not yet ahead — llama.cpp runs at roughly 78% of that machine's memory-bandwidth ceiling and caterpillar around 49%. The speculative path beats it on draftable output. We publish both numbers rather than the flattering one, and the gap is a kernel-efficiency problem we are still working on. If you want to see what your own machine does, the readiness check in setup gives you tokens per second for free — no credits, no benchmark suite needed.

lizard-native: the whole inference stack, running locally

lizardadmin — Fri, 17 Jul 2026 11:52:00 GMT

If Lizard is an inference provider, the fair question is which parts of the stack it actually owns. Here is the answer, top to bottom. [image: lizard-native-stack.png] The API layer speaks the OpenAI Chat Completions format. This is deliberate: it means the client libraries you already use work by changing a base URL, and it means you are not locked in — anything you build against Lizard runs against a hosted provider too, and vice versa. Scheduling and batching is where a lot of local runtimes lose time. The naive approach submits work to the GPU many times per token, and each submission costs you. lizard-native compiles the decoder graph into coalesced spans ahead of time and gets that down to roughly one submission per generated token. The KV cache is resident and its precision is yours to pick — f16, q8_0, q4_0. This is the setting nobody exposes and everybody should: it is a direct trade between context length and memory. A hosted provider makes that choice for you and you never see it. The kernels are where the model's weights get multiplied. These are quantization-aware — Q4_K, Q5_K, Q6_K — with the elementwise work fused into the matrix multiply so a token costs fewer passes over memory. The device layer is Direct3D 12, and the important word is resident: the weights are uploaded once and stay in GPU memory. Loading is the expensive part; after the first time, the shaders are cached on disk too, so a load is seconds rather than a minute. Why this matters for a developer: every one of those layers is a decision that determines your tokens per second, and on a hosted API all of them are made somewhere you cannot see or change. Locally, they are yours — the KV cache precision and the quantization level are literally settings in the app. The practical ceiling is memory bandwidth, not arithmetic. Generating a token means reading the whole model out of memory, so on most machines the model's size divided by your memory bandwidth is the speed limit. That is also why a smaller quantization is often faster, not just smaller.

Where do your tokens actually come from?

lizardadmin — Wed, 15 Jul 2026 11:18:00 GMT

When you call an LLM API, it is easy to assume the company you are calling is the one running the model. Often it isn't, and the difference matters more than it looks. [image: tokens-origin.png] An API router is a proxy. It owns no compute. Your request arrives, gets forwarded to whichever upstream provider it has a deal with, and the response comes back through it. Useful — one key, many models, automatic failover when an endpoint degrades. But every request takes an extra network hop, so median time-to-first-token is always worse than calling that same provider directly. Routers improve tail latency by rerouting around bad endpoints; they cannot improve the median. Nothing they do touches the decisions that actually determine speed. An inference provider owns the hardware. The same company controls the endpoint and the GPUs your tokens are computed on. They choose the batching strategy, the KV cache layout, the kernels, the quantization. Those choices are what your latency and throughput actually come from. There is a second, quieter difference. A router can only bind itself. If you sign a zero-retention agreement with a router, your request still lands at an upstream provider whose policies you never reviewed. The agreement doesn't travel with your data. Lizard is an inference provider where the provider is your own machine. The endpoint is 127.0.0.1:7817, the GPU is the one in your computer, and there is no upstream at all — no hop to add latency, no third party to have a retention policy, no agreement chain to audit. When you turn off your Wi-Fi, it keeps working. That is the honest test of the claim, and you can run it yourself. The trade is real and worth stating: you get one machine's worth of compute, not a fleet. A hosted provider will out-run your laptop on a 70B model. What you get instead is that nothing leaves the building. How do you tell which kind you are using? Read the docs. "Our clusters", "our GPUs" means a provider. "Access 200+ models from leading providers" means a router. In a DPA, any mention of sub-processors or third-party infrastructure partners is a router tell — a direct provider has no such chain.