RealFeedApp llama-sleep reduced model swap time to 1.7 seconds
llama-sleep preserves prompt cache in RAM instead of GPU, reducing model swap time from 10.3 to 1.7 seconds using transparent huge pages.
- Field
- applying language models
- What they did
- Developed a modification of llama.cpp called llama-sleep, which keeps a language model loaded in GPU memory by parking it in the computer's RAM instead of stopping it completely.
- Why it matters
- This allows switching between different language models in 1.7 seconds while keeping cached prompts intact, which is six times faster than previous methods and reduces GPU idle time.
#llama.cpp#llm#llm-swap#llama-sleep#gpu#memory
