Speculative decoding: why repetitive text suddenly runs faster
-
You may notice Caterpillar getting noticeably quicker on some kinds of output — lists, structured text, anything repetitive — and behaving normally on ordinary prose. That's deliberate.
The short version: generating a word normally means reading the whole model from memory once. If the next few words are predictable, Caterpillar drafts a guess and checks several words in a single pass instead of one pass each.
Two things worth knowing:
- It is lossless. You get exactly the same words you would have got anyway — it's verified against the real model, not an approximation. If a guess is wrong it's thrown away.
- It steps aside when it isn't helping. On normal prose, guesses rarely land, so it stops trying rather than making you pay for failed guesses.
So the honest summary is: sometimes much faster, never slower in a way you'd notice, never different output.
Anyone seen a big speed-up on a particular kind of prompt?
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login