vLLM: what it is, what it is for, and when to use something else
High-throughput inference server built around PagedAttention.
At a glance
| Capability | vLLM |
|---|---|
| Licence | Apache 2.0 |
| Implementation | Python + CUDA |
| Runs on | GPU only |
| Continuous batching | Yes |
| Paged attention | Yes |
| Tensor parallelism | Yes |
| Quantisation | AWQ, GPTQ, FP8, INT8 |
| OpenAI-compatible API | Yes |
vLLM is the reference open-source inference server for transformer models. Its central contribution is PagedAttention, which manages the KV cache in fixed-size blocks the way an operating system manages virtual memory pages. Because the cache no longer needs a contiguous reservation sized to the maximum sequence length, memory fragmentation drops sharply and far more sequences fit on a card at once.
On top of that sits continuous batching: rather than waiting for a batch to fill and finish together, vLLM admits new requests into the running batch as earlier sequences complete. Under real traffic — where prompt and completion lengths vary widely — this is the difference between a GPU that is busy and one that is waiting, and it is why throughput gains over naive serving are measured in multiples rather than percentages.
It supports tensor parallelism for models that exceed a single GPU, quantised weights via AWQ, GPTQ and FP8, and exposes an OpenAI-compatible HTTP API so client code is portable. The cost is operational: vLLM is a service you run, scale, monitor and keep current with a fast-moving upstream.
Where it stops being the right tool
GPU-only in practice, and operationally heavier than a local runner: you own the process, the memory tuning and the upgrade lane.
Frequently asked
- What is vLLM used for?
- Production serving of open models where throughput per GPU is the metric that matters, and for multi-GPU deployment of models too large for one card.
- What are the limitations of vLLM?
- GPU-only in practice, and operationally heavier than a local runner: you own the process, the memory tuning and the upgrade lane.
- Does vLLM need a GPU?
- Yes. vLLM targets GPU execution and is not a practical CPU inference path.
Compare with
- Ollama — The simplest way to run open models on your own machine.
- LM Studio — A desktop GUI for discovering, running and chatting with local models.
- llama.cpp — Portable C/C++ inference that runs open models almost anywhere.
- SGLang — Inference server optimised for structured generation and prefix reuse.
Project home: https://docs.vllm.ai