Lizard inference engineering: Batch means concurrent requests, not a compression sweep
-
Inference Engineering · Day 22 · Evening
The main trap in inference benchmarks is a bad unit label. If someone reads “batch size” and assumes tokens, sequence length, or a compression setting, the chart stops meaning what you think it means.
In Lizard’s HTTP benchmark, B=16 is deliberately narrow: sixteen concurrent client requests against one warm server. That keeps the concurrency axis honest. Quantization sweeps are tracked separately, so a run that changes model compression is not being mixed into a run that changes request pressure.
That distinction matters in practice. If you are tuning queueing, health checks, session reuse, or server warmup, you want the concurrency curve to move for one reason at a time. Otherwise a nice-looking result can hide whether the server handled more requests, a different model representation, or just a different benchmark mode.
I have found that the simplest check is to write the unit in the same sentence as the chart axis: concurrent requests, warm server, separate compression sweep. It removes a lot of debate later when the numbers get compared across runs, tools, or teams.
When your team says “batch,” what is the first thing you clarify from experience?
Engineering fact: In Lizard's HTTP benchmark, B=16 means sixteen requests in flight on one server; quantization sweeps are a separate benchmark dimension.
Lizard The AI Runtime You'll Own—Not Rent.
Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.
#llmbenchmark #inference #httpruntime #quantization #systemsengineering
<!-- lizard-marketing-slot:day-22-pm -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login