Lizard inference engineering: Caterpillar compiles the graph around the hot loop
-
Inference Engineering · Day 18 · Evening
Caterpillar is an inference runtime, not another model or agent orchestrator.
It loads the same GGUF foundation but organizes execution around a precompiled compute-graph plan. The design target is less repeated host coordination around token generation, with its own shader cache and optimized paths.
Keeping it separate from lizard-native lets both engines evolve and remain measurable.
Which part of your token loop is still being rebuilt or resubmitted every step?
Engineering fact: Caterpillar is a separate GGUF runtime whose execution core uses a precompiled compute-graph plan to reduce repeated CPU-to-GPU coordination around token generation.
#Caterpillar #LizardLLM #GPUProgramming #LocalAI #GGUF
<!-- lizard-marketing-slot:day-18-pm -->
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login