The main trap in inference benchmarks is a bad unit label. If someone reads “batch size” and assumes tokens, sequence length, or a compression setting, the chart stops meaning what you think it means.

In Lizard’s HTTP benchmark, B=16 is deliberately narrow: sixteen concurrent client requests against one warm server. That keeps the concurrency axis honest. Quantization sweeps are tracked separately, so a run that changes model compression is not being mixed into a run that changes request pressure.

That distinction matters in practice. If you are tuning queueing, health checks, session reuse, or server warmup, you want the concurrency curve to move for one reason at a time. Otherwise a nice-looking result can hide whether the server handled more requests, a different model representation, or just a different benchmark mode.

I have found that the simplest check is to write the unit in the same sentence as the chart axis: concurrent requests, warm server, separate compression sweep. It removes a lot of debate later when the numbers get compared across runs, tools, or teams.

When your team says “batch,” what is the first thing you clarify from experience?

Engineering fact: In Lizard's HTTP benchmark, B=16 means sixteen requests in flight on one server; quantization sweeps are a separate benchmark dimension.

Lizard The AI Runtime You'll Own—Not Rent.

Receive two professional Windows AI runtimes with lifetime updates. Run AI at native speed, keep every conversation private, and stay independent with intelligent hardware optimization and no cloud dependency.

Read the relevant Lizard page

#llmbenchmark #inference #httpruntime #quantization #systemsengineering

<!-- lizard-marketing-slot:day-22-pm -->