Lizard inference engineering: Let measured B16 and B8 results choose the lane
-
Inference Engineering · Day 14 · Morning
A fixed concurrency default ignores both hardware capacity and measured behavior.
Lizard treats B=16 as the throughput target and B=8 as the normal reduced-capacity lane. When both have completed HTTP evidence, the faster eligible primary lane wins. B=4 and B=1 are reserved for constrained fallback.
The policy turns a benchmark result into a serving decision without pretending every machine has the same memory budget.
Would your runtime choose the same batch lane on an integrated GPU and a discrete GPU?
Engineering fact: Lizard's adaptive concurrency policy chooses the faster completed B=16 or B=8 HTTP measurement when capacity permits; B=4 and B=1 remain fallbacks.
#LizardNative #LizardLLM #LocalAI #GGUF #InferenceEngineering
<!-- lizard-marketing-slot:day-14-am -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login