Lizard inference engineering: Separate user-visible speed from native decode speed
-
Inference Engineering · Day 13 · Morning
Decode speed and user-visible speed answer different operational questions, and serving stacks get into trouble when they are treated as the same metric.
Lizard Chat and native HTTP responses keep both numbers in view: the worker’s decode-only token rate, and the end-to-end rate the user actually experiences. That split matters because prompt handling, scheduling, transport, and response assembly all happen outside the decode loop. If you only watch decode tok/s, it is easy to miss where latency is really being spent.
In practice, the decode number is still useful. It tells you whether the kernel is making progress once the worker is running. But the user-visible number is the one that reflects the full request path. When those two move together, the serving path is usually healthy. When they diverge, the gap is a clue: session setup may be too expensive, prefill may be creeping up, or the handoff between worker and client may be doing more work than expected.
That distinction is especially useful in local runtimes, where overheads can dominate short requests. A model can look fast in isolation and still feel sluggish in chat because the request spends too much time before the first token or after the last one. Separating the metrics makes that visible without arguing over which number is “real.”
I’d rather see both than compress them into one optimistic line item.
Which part of your serving path only showed up after you split decode tok/s from user-visible tok/s?
Engineering fact: Lizard Chat and native HTTP responses track end-to-end user-visible token rate separately from the worker's decode-only token rate.
Lizard The AI Runtime You'll Own—Not Rent.
Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.
#LocalAI #LLMOps #Inference #TokenThroughput #SystemsEngineering
<!-- lizard-marketing-slot:day-13-am -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login