What if one pass over the model weights could accept several tokens?

Caterpillar's speculative path drafts a short sequence, then verifies multiple candidates in a single weight pass. It is lossless: rejected drafts fall back to the normal path, so accepted output remains the output the model would have produced.

On predictable or repetitive text, our tests reached up to 2× speed. On ordinary prose, the path steps aside rather than forcing a bad optimization.

Optimization should be conditional when the workload is conditional.

Engineering fact: Caterpillar's lossless speculative path can verify multiple drafted tokens per weight pass on predictable output and steps aside on ordinary prose.

Read the relevant Lizard page

#SpeculativeDecoding #CaterpillarEngine #LLMInference #LocalAI #LizardLLM

<!-- lizard-marketing-slot:day-05-pm -->