← Journal · · Note
What actually speeds up cold start for LLM servers: caches, snapshots and weight daemon
CNCF published an overview of approaches to fast startup for vLLM, SGLang and llm-d. The article covers weight and compilation caches, weight daemon with GPU memory reuse via CUDA IPC, sleep/wake and snapshots of initialized engines. Plus a breakdown of what's already implemented versus what remains as proposals in Kubernetes and OCI.
A useful detail: snapshot doesn't necessarily eliminate weight loading. For example, vLLM's experimental snapshot saves initialization but reloads weights from scratch; currently works on the same machine, single NVIDIA GPU, and TP=1.
A good map of the topic for those working on serving and autoscaling: what we preserve between restarts and which startup stages still cost us.