How to self-host an LLM

Before you rent or buy anything, do the arithmetic. A model needs memory for its weights and, on top of that, a KV cache that grows with every token of every request in flight. Most "it crashed on startup" reports are a budget that was never written down.

Weights cost about two bytes per parameter in 16-bit, one byte in 8-bit and half a byte in 4-bit. The KV cache costs, per token, 2 (keys and values) x layers x key-value heads x head dimension x bytes per value. Read those numbers from the model's config.json. For Qwen3-8B the file lists 36 layers, 8 key-value heads and a head dimension of 128, in bfloat16. That gives 2 x 36 x 8 x 128 x 2 = 147,456 bytes, about 144 KiB per token.

Now apply it. The model card gives 8.2 billion parameters, so the 16-bit weights are about 16.4 GB. One sequence at 32,768 tokens needs 147,456 x 32,768 bytes, about 4.5 GiB of cache. On a 24 GB card, vLLM's default --gpu-memory-utilization of 0.92 leaves about 22 GB, so roughly 5.7 GB is left after the weights. That is an upper bound of about 38,000 tokens in flight across all users, before activations and overhead. Eight users at 4,000 tokens each fits; one user at 40,000 tokens does not.

vLLM runs on Linux. Install the NVIDIA driver with your distribution's package manager first, then Docker, then the NVIDIA Container Toolkit. The toolkit is what lets a container use the GPU.

Once the toolkit is installed, register it with Docker and restart the daemon. Then run the sample workload from the NVIDIA documentation. If it prints the usual GPU table, the host is ready. If it fails, fix that now; every later error will look like a vLLM problem when it is a driver or runtime problem.

Pick a model whose licence you have read. Qwen3-8B is labelled apache-2.0 on its Hugging Face card, which allows commercial use with notice and attribution. If you are choosing between several candidates, do that properly first: see how to evaluate an open-source LLM.

Download the weights before you start the server. The vLLM troubleshooting page says that a hang during download is fixed by downloading with the hf command first and passing the local path to vLLM, and that loading from a network filesystem can hang, so use a local disk. Write down the commit hash shown on the model's "Files and versions" page and keep it with your notes. If you later serve straight from the Hub, the --revision flag accepts a branch, tag or commit id, so you can pin it there.

Record a checksum manifest for the files you downloaded. It costs seconds and lets you prove later that the weights on the server are the weights you approved.

This command follows the Docker example in the vLLM documentation and adds four things: a pinned image tag, a localhost-only port binding, a restart policy and an API key. The docs' own example uses latest; on 6 October 2026 the Docker Hub tags for vllm/vllm-openai follow the pattern v0.31.0-cu129, and PyPI lists 0.31.0 as the newest release. Check the tags page when you run this and pin whichever version you intend to test.

The -p 127.0.0.1:8000:8000 binding matters. Docker's documentation says ports published without an address are published on all host addresses, so a bare -p 8000:8000 can expose the API beyond the host. Binding to 127.0.0.1 keeps it private until your proxy is ready. --ipc=host gives the container the shared memory that tensor-parallel inference uses. --restart unless-stopped brings the server back after a reboot unless you stopped it on purpose.

--served-model-name sets the name clients must send, so you can swap the underlying model later without changing every client. --max-model-len caps context length; set it to what you actually need, because it directly limits cache use. --api-key makes the server require a key, but only on the /v1, /v2 and /inference paths. Step 6 deals with the rest.

Query the server from the host with curl before anything else touches it. The model list confirms the served name; the chat request confirms the model actually generates. The vLLM quickstart shows the same two calls for the chat and completions endpoints on port 8000.

Then send a request without the Authorization header. The /v1 paths should refuse it. If they do not, the key was not applied and you must fix that before the next step.

The vLLM security page is direct about this. The --api-key flag only covers paths under /v1, /v2, /inference and /cohere, and many other sensitive endpoints sit on the same server without authentication. It recommends a reverse proxy such as nginx, Envoy or a Kubernetes Gateway that allowlists only the endpoints you want to expose and blocks the rest. It also says nodes should sit on a dedicated, isolated network with all incoming connections blocked except the API port.

Caddy does this in a few lines and obtains certificates automatically for a public domain name. The Caddyfile below sends /v1 traffic to vLLM and returns 404 for everything else, including /metrics and /health. flush_interval -1 switches off response buffering so streamed tokens reach the client as they are produced.

If the server has no public DNS name, automatic certificates will not work. Either use an internal certificate authority with your proxy, or give vLLM its own certificate with --ssl-certfile and --ssl-keyfile and keep it on a private network. Either way, open the firewall to the proxy port only.

From the host, /health answers whether the server is up and /metrics returns Prometheus-format metrics. Both are blocked at the proxy by design, so scrape them from inside the host or the private network.

Four metrics tell you most of what you need. vllm:num_requests_running and vllm:num_requests_waiting show load and queueing. vllm:kv_cache_usage_perc shows how full the cache is; when it sits near the top, requests get preempted and latency climbs. vllm:time_to_first_token_seconds is a histogram of how long users wait for the first word. Alert on a growing waiting queue and on cache usage staying high, rather than on a single spike.

Then load it. The vLLM benchmark command sends a realistic request mix and reports time to first token, time per output token, inter-token latency and throughput. Run it with a dataset that looks like your traffic, not a toy one, and keep the output with your sizing notes.

When the cache is too small for the load, vLLM preempts and recomputes requests, which raises latency. Its optimisation guide lists four levers: raise --gpu-memory-utilization to give the cache more room; lower the maximum number of concurrent sequences or batched tokens to need less cache; raise --tensor-parallel-size to shard the weights across GPUs so each has more memory left for cache; or use pipeline parallelism to spread layers across GPUs. More parallelism adds synchronisation cost, so it is not free.

Change one thing at a time and rerun the same benchmark. Lowering --max-model-len is the cheapest lever if your users do not need long contexts. Moving to a quantised checkpoint is the next one, but evaluate the quantised model on your own tasks first, because quality and refusal behaviour can shift.

Add a second GPU when a single card cannot hold your weights and cache at the context you need, not because the first benchmark looked slow. Two cards on one machine use --tensor-parallel-size 2. Multi-node serving is a different project, and the security page warns that traffic between nodes is unencrypted by default and must stay on an isolated network.

vLLM ships often; PyPI shows 0.31.0 released on 5 October 2026. Every new version, new image tag, new quantisation and new model revision can change behaviour. The troubleshooting page even has a flag for one such case: --generation-config vllm reverts generation defaults to vLLM's neutral ones when quality seems to have changed after switching models.

Run the candidate on a second port, say 8001, with the same flags and a different container name. Replay a fixed set of your own prompts against old and new, compare answers and latency, and only then change the proxy to point at the new port. Keep the old container stopped but not deleted for a few days. Rolling back is then one line in the Caddyfile and caddy reload.

Write each change down: image tag, model revision, flags, date, who approved it. That record is also what you will want if anyone asks what was running on a given day. For a fuller regression routine, read how to validate your AI.