Load models JIT, unload them immediately
A 32GB GPU is a shared resource. The discipline is not "which model to run" but "how to load, serve, and evict models so VRAM stays free for the next task."
- problem
- Running multiple local LLMs on a single GPU leads to VRAM exhaustion, model-swap latency, and a GPU that is permanently half-occupied by a model nobody is using.
- scope
- A JIT loading discipline for local model hosting: load on request, serve, unload on idle, and keep VRAM free for the next task rather than pinning a resident model.
- environment
- A workstation with an RTX 5090 (32GB) running LM Studio and MLX on macOS, with an agent that dispatches coding, vision, and general tasks to model-specific endpoints.
Assumptions
- The GPU has enough VRAM for one model at a time but not two large models simultaneously.
- Model loading time (seconds to tens of seconds) is acceptable for non-interactive batch tasks but not for latency-sensitive interactive chat.
- An orchestrator can track model load state and route requests to the correct endpoint or trigger a load.
Limitations
- JIT loading adds cold-start latency; interactive use cases may need a resident model and a different strategy.
- VRAM fragmentation after repeated load/unload can reduce effective capacity until the process is restarted.
- This is a personal discipline, not a multi-tenant inference server design.
Table of contents 4 sections
Resident models are a tax
The default instinct is to keep a model loaded so the first request is fast. On a 32GB card, a 27B model in 2-bit quantization occupies most of the VRAM. A resident model means the GPU cannot serve a vision model, cannot run a quick embedding job, and cannot load a different coding model without an explicit eviction. The "fast first request" tax is paid in permanent VRAM occupation.
The JIT discipline inverts this: the model loads on request, serves the task, and unloads on idle. The cold-start cost is seconds for a local model on NVMe. For batch tasks (generate, analyze, summarize) that latency is acceptable. For interactive chat, a smaller resident model (7B class) can coexist with the JIT loading of larger models for specific tasks.
Unload aggressively, not politely
A model that stays loaded "in case the next request comes" is the root cause of VRAM exhaustion. The unload should be immediate after the task completes, not after a generous idle timeout. If the next task needs the same model, loading it again is cheaper than the OOM crash that happens when two models fight for VRAM.
The practical signal is: did the task complete? Then unload. Do not wait for idle. The orchestrator tracks which model is loaded; a new task that needs a different model triggers unload-then-load. If the new task needs the same model, the loaded instance is reused. This is the same pattern as a connection pool with a max size of one.
# load model for taskcurl -s http://localhost:1234/v1/models -X POST \ -d '{"model": "bonsai-27b-2bit"}' # serve the requestcurl -s http://localhost:1234/v1/chat/completions \ -d '{"prompt": "..."}' > result.json # unload immediatelycurl -s http://localhost:1234/v1/models -X DELETE \ -d '{"model": "bonsai-27b-2bit"}'# VRAM is now freeRoute by task, not by model preference
The orchestrator should route by task type, not by a default model. A coding task routes to a coding-tuned model; a vision task routes to a vision model; a general task routes to a fast small model. The routing decision is separate from the loading decision: routing picks the model, loading manages VRAM.
This avoids the trap of "I have a 27B model loaded, let me use it for everything." A 27B model is not better at every task than a 7B model; it is slower and more expensive in VRAM. The task router should know the task type and the model strengths, and the loader should enforce the one-model-at-a-time VRAM constraint.
Watch for VRAM fragmentation
Repeated load and unload can fragment VRAM the same way repeated malloc and free fragments heap. A model that loaded in 4 seconds at startup may take longer or fail after dozens of load/unload cycles because the allocator cannot find a contiguous block. The signal is load time increasing over the session.
The practical mitigation is to restart the inference server periodically, or when load time exceeds a threshold. This is not elegant, but it is honest: the inference server is a long-running process that accumulates allocator state, and a periodic restart is the simplest way to reset it.
- field-observed 32GB VRAM under multi-model load
Loading a 27B model for a code task, then a vision model for analysis, then unloading both kept VRAM below 4GB idle and avoided the OOM crashes that occur when models are pinned resident.
- reference LM Studio documentationopen ↗
Reference for the local inference server, model loading API, and VRAM management.