A model that almost fits can create the worst failure mode: it runs, but it thrashes.

On a tight GPU budget, repeatedly evicting and re-uploading layers can reduce decode to a crawl. A single “GPU or CPU” decision is too coarse.

Lizard plans prefill and decode independently. It can keep GPU-assisted prompt processing while moving decode to a bounded, stable path when full residency is unsafe.

The engineering lesson: split the phases before you split the blame.

Engineering fact: When a model exceeds the usable GPU weight budget, Lizard can keep GPU-assisted prompt processing and use bounded hybrid decode instead of repeated eviction.

Read the relevant Lizard page

#VRAM #LLMInference #GPUPerformance #LizardNative #LocalAI

<!-- lizard-marketing-slot:day-02-pm -->