Why does vLLM use all my GPU memory even with no requests?

By design. vLLM pre-allocates the PagedAttention KV cache at startup, claiming everything up to `gpu_memory_utilization` (0.9 by default) after weights and profiled activations. A near-full GPU at idle is normal and is what lets vLLM batch aggressively without fragmentation: lower the fraction if you need to share the card.

What should I set gpu_memory_utilization to in vLLM?

On a dedicated GPU, 0.90 to 0.95: high enough that the KV cache avoids preemption, with margin for CUDA context and NCCL buffers. On a shared GPU, divide by the number of resident processes, because the value is a fraction of total capacity rather than of free memory, so two instances at 0.9 will always OOM.

How do I fix "The model's max seq len is larger than the maximum number of tokens that can be stored in KV cache"?

Set `--max-model-len` to the longest sequence you actually serve rather than the model's native context window, which is often 128K. If the GPU has headroom you can instead raise `--gpu-memory-utilization`; if neither is enough, add tensor parallelism or use `--kv-cache-dtype fp8` to halve the per-token cache cost.

Does quantization fix vLLM out-of-memory errors?

Only partly. AWQ, GPTQ and FP8 shrink the weights, which frees memory for the KV cache and can move a model onto fewer GPUs, but the KV cache itself is unchanged and scales with `max_model_len` and concurrency. If your OOM message is about KV cache tokens, quantizing the weights alone may not clear it.

What does "preempted by PreemptionMode.RECOMPUTE" mean in vLLM?

The scheduler ran out of free KV blocks and evicted an in-flight sequence, discarding its cache to be recomputed when it is rescheduled. Requests still complete correctly, but latency and throughput suffer. Reduce `--max-num-seqs`, give the KV cache more memory, or add GPUs with tensor parallelism.

Error reference

vLLM Out of Memory: Fixing Startup and Runtime OOM

Updated July 27, 2026

In short

vLLM does not allocate memory on demand: at startup it loads the weights, profiles a peak-activation forward pass, and then claims *all* remaining memory up to gpu_memory_utilization (default 0.9) as a paged KV cache. So a vLLM OOM is usually a budgeting error, not a leak. The two fixes that resolve most cases: lower --max-model-len so the KV cache needed for one sequence fits, and set --gpu-memory-utilization to match what the GPU actually has spare. If the weights themselves are too large, shard with --tensor-parallel-size or serve a quantized checkpoint.

start here

vllm serve <model> --max-model-len 8192 --gpu-memory-utilization 0.90

Diagnosis

Causes at a glance

Cause	How to recognize it	Fix
max_model_len larger than the KV cache can hold	"The model's max seq len (N) is larger than the maximum number of tokens that can be stored in KV cache"	Lower `--max-model-len`, or raise `--gpu-memory-utilization` if headroom exists
gpu_memory_utilization set too high for a shared GPU	CUDA OOM during weight loading or profiling, another process visible in `nvidia-smi`	Lower `--gpu-memory-utilization`; it is a fraction of total VRAM, not of free VRAM
gpu_memory_utilization set too low	"No available memory for the cache blocks" at engine init	Raise it toward 0.90-0.95 on a dedicated GPU so the KV cache has room
Weights do not fit on one GPU	OOM while loading safetensors shards, before any cache profiling	Use `--tensor-parallel-size N`, or serve an AWQ/GPTQ/FP8 quantized checkpoint
Too much concurrency at runtime	Server starts fine, then logs preemption warnings under load	Lower `--max-num-seqs` and `--max-num-batched-tokens`

The counterintuitive part: vLLM pre-allocates

Most inference stacks grow their memory footprint as requests arrive. vLLM does the opposite. At engine start it loads the weights, runs a profiling forward pass to measure peak activation memory, and then reserves everything left below the gpu_memory_utilization fraction as a pool of fixed-size KV cache blocks. That pool is the whole point of PagedAttention: block-level allocation is what lets vLLM pack many sequences without per-request fragmentation.

Two consequences trip people up. First, nvidia-smi will show the GPU near-full immediately after startup even with zero traffic. That is expected, not a leak. Second, gpu_memory_utilization is a fraction of the GPU's *total* capacity, not of what is currently free. On a card already holding another process, 0.9 means "take 90% of the whole card", and the engine will OOM trying.

INFO ... Memory profiling takes 4.31 seconds
INFO ... the current vLLM instance can use total_gpu_memory (79.15GiB)
         x gpu_memory_utilization (0.90) = 71.23GiB
INFO ... model weights take 14.99GiB; non_torch_memory takes 0.63GiB;
         PyTorch activation peak memory takes 1.15GiB;
         the rest of the memory reserved for KV Cache is 54.46GiB

Step 1

Step 1: decide which OOM you have

Read where in the log the failure lands, because the two failure modes have different fixes.

"Engine core initialization failed" / a CUDA OOM traceback before the server binds its port is a startup budgeting problem: weights + activations + requested KV cache exceed the budget. A ValueError naming max seq len and the KV cache is the same class of problem, caught politely instead of as a hard OOM.

A server that starts, serves traffic, and then logs preemption warnings is not out of memory in the failure sense: vLLM is deliberately evicting and recomputing sequences because the KV pool is saturated. That is a throughput and latency problem, and the fix is concurrency limits or more KV memory, not a restart.

# startup, caught as a ValueError:
ValueError: The model's max seq len (32768) is larger than the maximum
number of tokens that can be stored in KV cache (12464). Try increasing
`gpu_memory_utilization` or decreasing `max_model_len` when initializing
the engine.

# runtime, under load:
WARNING ... Sequence group ... is preempted by PreemptionMode.RECOMPUTE mode
because there is not enough KV cache space. This can affect the end-to-end
performance. Increase gpu_memory_utilization or tensor_parallel_size to
provide more KV cache memory.

Step 2

Step 2: set max_model_len to what you actually serve

By default vLLM uses the context length declared in the model config, which for current open-weight models is often 128K or more. The engine must be able to hold at least one sequence of that length in the KV cache, so a model whose native context is 128K can fail to start on a GPU that would happily serve it at 16K.

The KV cache requirement scales linearly with max_model_len, so this is the single highest-use knob. Set it to the longest request you genuinely serve plus your maximum generation length, not to the model's theoretical maximum. Halving it roughly doubles the number of concurrent sequences you can hold.

--kv-cache-dtype fp8 is the other lever on the same term: it halves the bytes per cached token, with a small accuracy cost, and is supported on hardware with FP8 tensor cores.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8

Step 3

Step 3: tune gpu_memory_utilization in the right direction

This parameter moves in both directions, which is why advice about it seems contradictory. On a dedicated GPU, raise it: 0.90 is the default and 0.92-0.95 is usually safe, giving the KV cache more room and eliminating preemption. Leave some margin: CUDA context, NCCL buffers and any non-PyTorch allocations live outside the accounted budget, and pushing to 0.99 makes the engine fragile.

On a shared GPU, lower it. Because the fraction is of total capacity, running two vLLM instances at 0.9 each is an immediate OOM; two instances need roughly 0.45 each, minus overhead. The same applies when a training job, another framework, or a display server is already resident.

--swap-space (CPU memory used to swap out preempted sequences, 4 GiB per GPU by default) affects host RAM rather than VRAM, and is worth reducing in memory-limited containers where the host is also constrained.

# two instances sharing one 80GB GPU
CUDA_VISIBLE_DEVICES=0 vllm serve model-a --gpu-memory-utilization 0.45 --port 8000 &
CUDA_VISIBLE_DEVICES=0 vllm serve model-b --gpu-memory-utilization 0.45 --port 8001 &

Step 4

Step 4: cap concurrency, then scale out

--max-num-seqs caps how many sequences run in one batch (256 by default) and --max-num-batched-tokens caps the tokens processed per scheduler step. Lowering them reduces peak KV pressure and preemption at the cost of throughput; it is the correct fix when the server starts cleanly but thrashes under load.

--enforce-eager disables CUDA graph capture. CUDA graphs cut per-step launch overhead and meaningfully improve decode latency, but capturing them costs extra memory. Setting it is a legitimate last resort to squeeze an engine onto a card, with a real latency penalty: do not set it by default.

When the weights themselves are the problem, scale out or scale down. --tensor-parallel-size N splits every layer across N GPUs on one node; N must divide the number of attention heads. --pipeline-parallel-size splits by layer and is the multi-node option. Quantization is the alternative: AWQ and GPTQ checkpoints hold 4-bit weights, and FP8 works on Hopper-class and newer hardware with a smaller accuracy cost. Note that quantization shrinks weights, not the KV cache: a quantized model with an unchanged --max-model-len can still fail on the same error.

# four GPUs, one node
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192

# or fit it on fewer GPUs with 4-bit weights
vllm serve TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ --quantization awq

Verifying the budget instead of guessing

The startup log prints the full breakdown: total GPU memory, the utilization fraction, weight size, non-torch memory, peak activation memory, and the remainder given to the KV cache. It also reports the resulting concurrency: how many sequences of max_model_len fit at once. If that number is below 1, the engine cannot start; if it is 1.2, you have a single-user server that will preempt constantly.

Treat that line as the acceptance test for any change you make. Raising gpu_memory_utilization by 0.03 or halving max_model_len should move it measurably; if it does not, you are tuning the wrong term.

INFO ... GPU KV cache size: 445,072 tokens
INFO ... Maximum concurrency for 16,384 tokens per request: 27.16x

Beyond the one-off fix

Self-hosted inference has no invoice, which is exactly why it escapes cost governance: a vLLM replica that preempts under load is a capacity decision no one can price without knowing which agents and services are calling it and how many tokens each consumed. Nexus records agent sessions and per-call token usage across every model endpoint, local or hosted, so the traffic hitting your vLLM server is attributable to named agents and teams, and you can see whether a saturated KV cache came from a legitimate workload or from an unmonitored agent quietly pointed at the internal endpoint to dodge API budgets.

Explore Swfte Nexus

Common questions

Why does vLLM use all my GPU memory even with no requests?: By design. vLLM pre-allocates the PagedAttention KV cache at startup, claiming everything up to `gpu_memory_utilization` (0.9 by default) after weights and profiled activations. A near-full GPU at idle is normal and is what lets vLLM batch aggressively without fragmentation: lower the fraction if you need to share the card.
What should I set gpu_memory_utilization to in vLLM?: On a dedicated GPU, 0.90 to 0.95: high enough that the KV cache avoids preemption, with margin for CUDA context and NCCL buffers. On a shared GPU, divide by the number of resident processes, because the value is a fraction of total capacity rather than of free memory, so two instances at 0.9 will always OOM.
How do I fix "The model's max seq len is larger than the maximum number of tokens that can be stored in KV cache"?: Set `--max-model-len` to the longest sequence you actually serve rather than the model's native context window, which is often 128K. If the GPU has headroom you can instead raise `--gpu-memory-utilization`; if neither is enough, add tensor parallelism or use `--kv-cache-dtype fp8` to halve the per-token cache cost.
Does quantization fix vLLM out-of-memory errors?: Only partly. AWQ, GPTQ and FP8 shrink the weights, which frees memory for the KV cache and can move a model onto fewer GPUs, but the KV cache itself is unchanged and scales with `max_model_len` and concurrency. If your OOM message is about KV cache tokens, quantizing the weights alone may not clear it.
What does "preempted by PreemptionMode.RECOMPUTE" mean in vLLM?: The scheduler ran out of free KV blocks and evicted an in-flight sequence, discarding its cache to be recomputed when it is rescheduled. Requests still complete correctly, but latency and throughput suffer. Reduce `--max-num-seqs`, give the KV cache more memory, or add GPUs with tensor parallelism.