How many CPU threads should Caterpillar use?
-
Short answer: leave it alone — the default is now your physical core count, which measured fastest.
Longer answer, because it surprises people: using every logical thread is usually slower, not faster. On a 4-core/8-thread laptop, 4 threads measured about 9% faster than 8, with steadier timings.
The reason is that generating text is limited by how fast the machine can read the model from memory, not by how much arithmetic it can do. Extra threads add contention without adding memory bandwidth.
If you want to experiment,
CATERPILLAR_CPU_THREADSoverrides it. If you try it, please post both numbers and your CPU — one result on its own is hard to learn from, since thermal state moves these numbers around a lot.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login