Lizard inference engineering: Do not ship an unproven fast path
-
Inference Engineering · Day 7 · Evening
We implemented an AVX-512 VNNI decode kernel and chose not to enable it by default.
The instruction is promising on supported Intel and AMD CPUs. But on the hardware available for validation, the measured result remained inside ordinary run-to-run noise.
Caterpillar ships the proven path as default and keeps VNNI behind an explicit experiment flag.
Engineering credibility sometimes means declining to claim a speedup.
Engineering fact: Caterpillar's AVX-512 VNNI kernel remains opt-in because it measured within normal run-to-run noise on available test hardware.
#AVX512 #CPUOptimization #Benchmarking #CaterpillarEngine #LizardLLM
<!-- lizard-marketing-slot:day-07-pm -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login