Running a model that's bigger than your graphics card
-
You don't need a card big enough to hold the whole model.
When a model doesn't fit, Lizard has two honest options and it will tell you which one it took:
- Stream the weights through the GPU. The card still does the work, but it reads parts of the model as it goes. Slower than fully resident, but it's still a real GPU result.
- Hand it to Caterpillar, which is built for CPU and shared graphics.
What it deliberately will not do is quietly pin an oversized model into shared memory and let it swap — that produces a machine that feels broken for no visible reason. If your model is too big for comfort, Lizard now says so and picks a supported path instead.
Rule of thumb: a Q4 model needs roughly 60% of the download size in free graphics memory to stay fully resident.
Anyone running a large model on a small card? What did you settle on?
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login