Lizard inference engineering: B16 is the throughput lane, not a single-user promise
-
Inference Engineering · Day 11 · Evening
I think one thing that's easy to misunderstand in LLM benchmarks is tokens/sec.
Our latest benchmark showed 40.7 tok/s for Lizard versus 21.8 tok/s for llama.cpp, using the exact same Llama 3.2 3B Q4_K_M GGUF model.
The important part is that this is aggregate throughput with 16 concurrent requests (B=16) over multiple real HTTP runs. It's not the speed of a single chat response.
Per-user latency and aggregate throughput answer different questions, and I think we should be much clearer about which one we're talking about when comparing inference engines.
Whenever I see a tokens/sec number now, my first question is: Is that per response, or total throughput?
Engineering fact: On one verified Llama 3.2 3B Q4_K_M run series, Lizard measured a 40.736 tok/s median at B=16 versus 21.755 tok/s for llama.cpp using the same physical GGUF; the result is aggregate HTTP throughput on that test system.
#LizardLLM #LLMInference #Benchmarking #LocalAI #PerformanceEngineering
<!-- lizard-marketing-slot:day-11-pm -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login