I think one thing that's easy to misunderstand in LLM benchmarks is tokens/sec.

Our latest benchmark showed 40.7 tok/s for Lizard versus 21.8 tok/s for llama.cpp, using the exact same Llama 3.2 3B Q4_K_M GGUF model.

The important part is that this is aggregate throughput with 16 concurrent requests (B=16) over multiple real HTTP runs. It's not the speed of a single chat response.

Per-user latency and aggregate throughput answer different questions, and I think we should be much clearer about which one we're talking about when comparing inference engines.

Whenever I see a tokens/sec number now, my first question is: Is that per response, or total throughput?

Engineering fact: On one verified Llama 3.2 3B Q4_K_M run series, Lizard measured a 40.736 tok/s median at B=16 versus 21.755 tok/s for llama.cpp using the same physical GGUF; the result is aggregate HTTP throughput on that test system.

Read the relevant Lizard page

#LizardLLM #LLMInference #Benchmarking #LocalAI #PerformanceEngineering

<!-- lizard-marketing-slot:day-11-pm -->