LLM runtimes and inference servers compared
These tools split cleanly into two groups that are often compared but rarely interchangeable: runners built to get a model working on one machine, and servers built to keep a GPU saturated under concurrent traffic. Choosing across the boundary is the most common mistake.
Local runners
Optimised for getting a model running on a single machine with minimal setup. No continuous batching, so throughput under parallel load is limited by design — that is a trade, not a defect.
OllamaThe simplest way to run open models on your own machine.MIT · CPU or GPULM StudioA desktop GUI for discovering, running and chatting with local models.Proprietary (free for personal use) · CPU or GPUllama.cppPortable C/C++ inference that runs open models almost anywhere.MIT · CPU or GPU
Inference servers
Built for serving. Continuous batching and paged attention keep the GPU busy across many concurrent requests, and tensor parallelism lets a model span several cards. You own the process and its operations.
vLLMHigh-throughput inference server built around PagedAttention.Apache 2.0 · GPU onlySGLangInference server optimised for structured generation and prefix reuse.Apache 2.0 · GPU onlyText Generation Inference (TGI)Hugging Face's production inference server.Apache 2.0 · GPU only