How do I fix CUDA out of memory in PyTorch?

Work down the list in order of impact: kill stale processes holding the GPU, halve the micro-batch size and use gradient accumulation to keep the effective batch, enable bf16 autocast and gradient checkpointing, then set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` if the error shows a large reserved-but-unallocated figure. Only after those should you reach for sharding or CPU offload.

Does torch.cuda.empty_cache() fix CUDA out of memory?

Rarely. It returns cached blocks that PyTorch was already free to reuse, so it does not create new capacity inside your own process, and it never frees a tensor that is still referenced. It is useful mainly for handing memory to a different process or library in the same container, and it costs a synchronisation, so calling it in a training loop just slows things down.

Why does CUDA OOM happen even though nvidia-smi shows free memory?

Because the allocator needs a contiguous block of the requested size, not just that many free bytes in total. Fragmentation in PyTorch's caching allocator can leave several GiB free while a single 2 GiB request still fails. Setting `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is the standard fix; `torch.cuda.memory_summary()` will confirm the fragmentation.

How much GPU memory do I need to fine-tune a 7B model?

Full fine-tuning in mixed precision with Adam needs roughly 16 bytes per parameter for weights, gradients and optimizer state: about 112 GB for 7B, before activations, so it requires multiple GPUs or ZeRO/FSDP sharding. LoRA or QLoRA with a 4-bit base model, gradient checkpointing and a small micro-batch brings the same job onto a single 24 GB card.

What does "Tried to allocate" mean in the CUDA OOM error?

It is the size of the single allocation that failed, not the total your job needs. A small "Tried to allocate" value alongside a nearly full GPU means you are just over the limit and a modest batch reduction will clear it; a multi-gigabyte value usually points at one oversized tensor, such as logits over a large vocabulary for a long sequence.

Error reference

CUDA Out of Memory (PyTorch): What It Means and How to Fix It

Updated July 27, 2026

In short

torch.cuda.OutOfMemoryError means PyTorch asked the CUDA driver for a block of VRAM and there was not enough contiguous free memory to satisfy it. The message tells you exactly how much was requested, how much is free, and how much PyTorch is holding: read those three numbers first. The primary fixes, in order of effect: reduce batch size (using gradient accumulation to keep the effective batch), enable mixed precision (bf16) and gradient checkpointing, and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True when reserved memory is much larger than allocated memory.

start here

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python train.py

Diagnosis

Causes at a glance

Cause	How to recognize it	Fix
Batch or sequence length too large	OOM on the first forward/backward pass, free memory near zero	Halve the micro-batch size and recover the effective batch with gradient accumulation
Activations stored for backward	Forward pass succeeds, OOM lands inside `loss.backward()`	Enable gradient checkpointing and bf16/fp16 autocast
Allocator fragmentation	"reserved by PyTorch but unallocated" is large while the request is small	Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
KV cache growth during inference	Works for short prompts, OOMs as generation length or concurrency rises	Cap max new tokens and concurrent requests, or use a paged-attention server
Another process holds the GPU	`nvidia-smi` shows a second PID (dead notebook, prior run) using VRAM	Kill the stale process; PyTorch cannot use memory another process reserved

What the CUDA OOM message is actually telling you

The exception is not generic. It is a memory report. Modern PyTorch prints the size of the failed request, the total capacity of the device, how much is free, how much the process holds, and the split between memory PyTorch has handed to tensors ("allocated") and memory PyTorch is holding in its caching allocator but not using ("reserved but unallocated").

Those last two numbers decide your fix. If allocated is close to capacity, you genuinely need less memory: smaller batch, lower precision, checkpointing, or sharding. If allocated is comfortably below capacity but reserved is huge and a small request still failed, the problem is fragmentation, not volume: the allocator has free memory, just not in a contiguous block of the right size.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 23.65 GiB of which 1.31 GiB is free.
Process 21458 has 22.33 GiB memory in use. Of the allocated memory
20.71 GiB is allocated by PyTorch, and 1.09 GiB is reserved by PyTorch
but unallocated. If reserved but unallocated memory is large try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Step 1

Step 1: confirm nothing else is holding the GPU

Before changing any hyperparameter, check that your process is the only one on the device. Crashed training runs, orphaned Jupyter kernels, and a second container scheduled onto the same GPU all keep VRAM reserved until the process exits. A GPU that reports 20 GiB in use with no running job of yours is a stale process, not a model that got bigger.

Note that memory held by another process is invisible to PyTorch: it cannot be freed from inside your script, and no allocator setting will recover it.

nvidia-smi
# or, just the processes and their memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
kill -9 <stale-pid>

Step 2

Step 2: cut activation memory with batch size and accumulation

Activation memory scales roughly linearly with batch size and with sequence length, and it is usually the largest term in a training run after the optimizer states. Halving the micro-batch is the fastest way to get under the limit.

You do not have to change the effective batch size to do it. Gradient accumulation runs several small micro-batches and only steps the optimizer after N of them, so the gradient statistics stay identical to the large batch while peak memory drops to that of one micro-batch.

accum_steps = 8  # micro_batch 2 x 8 == effective batch 16

for i, batch in enumerate(loader):
    with torch.autocast("cuda", dtype=torch.bfloat16):
        loss = model(**batch).loss / accum_steps
    loss.backward()
    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

Step 3

Step 3: enable mixed precision and gradient checkpointing

Mixed precision stores activations in 16-bit instead of 32-bit, roughly halving activation memory and speeding up matmuls on tensor cores. Prefer bfloat16 on Ampere-generation GPUs and newer. It has the same exponent range as fp32, so it needs no loss scaler. On older cards use float16 with torch.amp.GradScaler to avoid gradient underflow.

Gradient checkpointing goes further: instead of keeping every intermediate activation for the backward pass, it keeps a few checkpoints and recomputes the rest. That trades roughly 20-30% extra compute for a large drop in activation memory, and it is often the difference between fitting and not fitting a long-context fine-tune.

The other big term is optimizer state. Adam in mixed-precision training keeps fp32 parameters plus two moment buffers: on the order of 16 bytes per parameter, independent of batch size. If you are OOM before the first step even at batch size 1, the optimizer is the thing to shrink: use an 8-bit optimizer, or shard the state across GPUs.

# Hugging Face Transformers
model.gradient_checkpointing_enable()
model.config.use_cache = False  # required: cache and checkpointing conflict

# TrainingArguments equivalent
# bf16=True, gradient_checkpointing=True,
# per_device_train_batch_size=2, gradient_accumulation_steps=8,
# optim="adamw_bnb_8bit"

Step 4

Step 4: fix fragmentation, and why empty_cache() rarely helps

PyTorch uses a caching allocator: when a tensor is freed, the block is kept in PyTorch's pool for reuse rather than returned to the driver. Over a run with varying shapes, that pool fragments into blocks that are individually free but collectively unusable for one large request, which is how a 2 GiB allocation fails on a device with 4 GiB free.

expandable_segments:True changes the allocator to use growable virtual memory segments instead of fixed-size cached blocks, which largely removes this class of failure. It is the first thing to try when reserved-but-unallocated memory is large. The older max_split_size_mb knob addresses the same problem more crudely by preventing large blocks from being split.

torch.cuda.empty_cache() only returns cached-but-unused blocks to the driver. It does not free any tensor that is still referenced, and the memory it releases is memory PyTorch would have reused anyway, so it almost never fixes an OOM inside a single process. It is genuinely useful in one case: handing VRAM back so a *different* process or library (for example a separate inference engine in the same container) can allocate it. If you want memory back, delete the references first and let Python collect them.

# environment, before the process starts:
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# in-process, when handing memory to another library:
del logits, hidden_states
import gc; gc.collect()
torch.cuda.empty_cache()
print(torch.cuda.memory_summary())  # full allocator breakdown

Inference OOM is a different problem: the KV cache

Training OOMs are about activations and optimizer state. Inference OOMs are almost always about the KV cache: the stored keys and values for every token generated so far. It grows linearly with sequence length and with the number of concurrent requests, which is why a server runs fine in testing and dies under load or on one long conversation.

Rough size: 2 (K and V) x layers x kv_heads x head_dim x tokens x bytes_per_element, summed across all in-flight sequences. Grouped-query attention shrinks the kv_heads term substantially; fp8 KV cache halves the bytes term.

The practical fixes are to cap max_new_tokens and the number of concurrent sequences, or to stop hand-rolling generation and use a server with paged attention and preemption built in. If the weights themselves do not fit, shard across GPUs with tensor parallelism, or offload layers to CPU/disk with device_map="auto": offload works but is dramatically slower, so treat it as a way to run a model at all rather than a way to serve it.

# weight-only sizing rule of thumb:
#   fp16/bf16 ~ 2 GB per billion parameters
#   int8      ~ 1 GB per billion parameters
#   int4      ~ 0.5 GB per billion parameters
# then add KV cache + activations on top.

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    dtype="bfloat16",
    device_map="auto",              # shard across visible GPUs, offload the rest
    max_memory={0: "22GiB", "cpu": "64GiB"},
)

Beyond the one-off fix

A GPU that is full is a capacity question before it is a code question: which job, which agent, and which team took the VRAM, and was that run worth the GPU-hours it burned before it crashed? Nexus captures agent sessions, the commands and training scripts they launch, and the token and compute cost attached to each, so a repeated OOM on a shared node resolves to a named workload and an owner rather than a scheduling mystery, and self-hosted GPU spend sits in the same cost view as API spend instead of being invisible because no invoice arrives for it.

Explore Swfte Nexus

Common questions

How do I fix CUDA out of memory in PyTorch?: Work down the list in order of impact: kill stale processes holding the GPU, halve the micro-batch size and use gradient accumulation to keep the effective batch, enable bf16 autocast and gradient checkpointing, then set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` if the error shows a large reserved-but-unallocated figure. Only after those should you reach for sharding or CPU offload.
Does torch.cuda.empty_cache() fix CUDA out of memory?: Rarely. It returns cached blocks that PyTorch was already free to reuse, so it does not create new capacity inside your own process, and it never frees a tensor that is still referenced. It is useful mainly for handing memory to a different process or library in the same container, and it costs a synchronisation, so calling it in a training loop just slows things down.
Why does CUDA OOM happen even though nvidia-smi shows free memory?: Because the allocator needs a contiguous block of the requested size, not just that many free bytes in total. Fragmentation in PyTorch's caching allocator can leave several GiB free while a single 2 GiB request still fails. Setting `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is the standard fix; `torch.cuda.memory_summary()` will confirm the fragmentation.
How much GPU memory do I need to fine-tune a 7B model?: Full fine-tuning in mixed precision with Adam needs roughly 16 bytes per parameter for weights, gradients and optimizer state: about 112 GB for 7B, before activations, so it requires multiple GPUs or ZeRO/FSDP sharding. LoRA or QLoRA with a 4-bit base model, gradient checkpointing and a small micro-batch brings the same job onto a single 24 GB card.
What does "Tried to allocate" mean in the CUDA OOM error?: It is the size of the single allocation that failed, not the total your job needs. A small "Tried to allocate" value alongside a nearly full GPU means you are just over the limit and a modest batch reduction will clear it; a multi-gigabyte value usually points at one oversized tensor, such as logits over a large vocabulary for a long sequence.