Advanced: the AVX-512 VNNI option, and why it's off
-
For anyone who likes the details.
Modern Intel (Ice Lake and newer) and AMD (Zen 4 and newer) processors have an instruction that does 8-bit integer maths faster. Caterpillar has a kernel that uses it, behind
CATERPILLAR_CPU_VNNI=1.It ships disabled, and the reason is worth stating plainly: on our test hardware it measured within run-to-run noise of the standard path. Not faster, not slower — indistinguishable. Shipping it on by default would be claiming a win we can't demonstrate.
It may well help on other chips, particularly ones with more memory bandwidth. If you try it:
CATERPILLAR_CPU_VNNI=1Please post your CPU and both numbers, run back-to-back. Alternating measurements matter here — machines get hotter as you test them, and that alone can swamp the difference.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login