Router Mode in llama.cpp: Finally, a Native Alternative to Ollama’s Model Switching

Local LLM deployment has always forced a uncomfortable choice: run one model per server instance and eat the memory overhead, or build fragile orchestration layers to switch between them. The latest llama.cpp update fundamentally changes this calculus. Router mode collapses what used to require multiple processes and manual coordination into a single, intelligent server instance that manages models on demand.

For developers who’ve been cobbling together solutions with llama-swap or settling for Ollama’s convenience at the cost of control, this is a watershed moment. But as with any infrastructure shift, the devil lives in the details, VRAM management, multi-GPU complexity, and the subtle differences from existing tools.

What Router Mode Actually Does

The core proposition is brutally simple: start llama-server once, then load and unload models dynamically as requests arrive. No more restarting the server. No more juggling multiple ports. The server automatically routes each request to the appropriate model based on the model field in your API call.

Previously, serving multiple models meant running llama-server --model model1.gguf on port 8080, llama-server --model model2.gguf on port 8081, and building a proxy layer to direct traffic. Each instance burned memory for its model and overhead. Router mode replaces this with a single parent process that spawns child server instances on demand.

The Ollama Comparison: Convergence and Divergence

The immediate reaction on developer forums has been predictable: “Finally, I get to ditch Ollama!” But the reality is more nuanced. Router mode brings Ollama-like functionality to llama.cpp’s lightweight, unopinionated core, but the implementation philosophies diverge significantly.

Ollama bundles models, inference engine, and management into a polished package with its own model registry and abstraction layers. Router mode gives you raw control. You get the models you have on disk, loaded with parameters you specify, managed through API calls or command-line flags. No hidden magic, but also no hand-holding.

The llama-swap Factor: Coexistence, Not Replacement

Router mode doesn’t obsolete llama-swap, it narrows its use case. The key architectural difference is scope: llama-swap orchestrates arbitrary inference engines (llama.cpp, vLLM, SGLang) as a universal proxy, while router mode works exclusively within llama.cpp’s ecosystem.

For users who downloaded models with the -hf switch, router mode offers zero-configuration discovery. The web UI automatically populates a dropdown of available models. Start the server, pick a model, and you’re running. Configuration happens through --models-preset ./my-models.ini, where you can specify per-model parameters like context size, GPU offloading, and CPU threads.

Under the Hood: Architecture and Trade-offs

The HTTP server architecture reveals why this matters for production systems. The server uses a slot-based concurrency model with continuous batching enabled by default. Each incoming request grabs a slot, gets batched with compatible requests, and executes through a unified llama_decode() call. Router mode extends this by adding a process management layer.

The request flow in router mode follows a specific path:

1. HTTP worker thread receives request with model field
2. server_routes delegates to server_models
3. server_models checks if model is loaded
4. If not, spawns child llama-server process with model
5. Forwards request and streams response back

The VRAM Management Elephant in the Room

The most sophisticated discussion around router mode centers on VRAM allocation, especially for multi-GPU systems. Developers on systems with PCIe bandwidth constraints quickly discover that spreading models across all GPUs isn’t optimal. Some models perform dramatically better when confined to specific GPUs, but manually managing this creates a configuration nightmare.

One developer described this as a “knapsack-style problem” where you must consider:
– Model sizes across different quantizations
– Tensor offloading strategies (especially for MoE models)
– Context length variations
– GPU affinity and PCIe topology
– Swap-in/swap-out priorities

Practical Implementation: Getting Started

The basic usage pattern is intentionally minimal. Start the server without specifying a model:

llama-server

Then send requests with the model specified in the payload:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-cool-model-v1-GGUF",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The first request loads the model, which can take seconds to minutes depending on size and hardware. Subsequent requests to the same model are near-instantaneous. The server automatically caches and manages models based on the LRU eviction strategy.

Limitations and Gotchas

Router mode is powerful but not a panacea. The child process architecture means each model load incurs full initialization overhead, including GPU memory allocation and tensor offloading. On constrained systems, loading a large model can take several minutes, during which other requests may queue.

The Bottom Line

Router mode represents llama.cpp maturing from a research tool into production-ready infrastructure. It eliminates the operational complexity that drove many users toward Ollama while preserving the raw performance and transparency that made llama.cpp dominant in the local LLM space.