Why is my pod OOMKilled when kubectl top shows usage below the limit?

kubectl top reports periodic samples, and the kill happens at the instant of an allocation spike between samples. Short-lived surges (large request bodies, batch loads, GC churn) can cross the limit without ever appearing in sampled metrics.

Should memory requests equal limits in Kubernetes?

For memory, it is a common and defensible default. Memory cannot be reclaimed like CPU, so scheduling a pod on less than it may use creates node pressure and eviction risk. Requests = limits gives that resource Guaranteed-style predictability.

Does an OOMKilled container restart automatically?

Yes, per the pod restartPolicy (Always for Deployments), and repeated kills put it into CrashLoopBackOff with growing delays. The restart does not fix anything: usage will climb back to the limit unless you change the limit or the app.

How much headroom should I leave between heap size and the container limit?

Around 20-25%. The runtime needs memory beyond the heap: thread stacks, code cache, off-heap and native buffers. A 2Gi container with a 1.5Gi heap is a reasonable starting ratio; tune from observed working-set data.

Can I see what the app was doing when it was OOMKilled?

Not from the kill itself: SIGKILL leaves no application trace. You reconstruct it from metrics and logs leading up to the kill, or configure the runtime ahead of time (e.g. JVM heap dumps, allocation profiling) so the next kill leaves evidence.

Error reference

OOMKilled in Kubernetes: Why Pods Run Out of Memory and How to Fix It

Updated July 27, 2026

In short

OOMKilled means the Linux kernel killed your container because it exceeded its memory limit, the pod shows exit code 137 with Reason: OOMKilled. The limit is a hard cgroup cap: the moment usage crosses resources.limits.memory, the OOM killer fires with no warning and no chance to clean up. Confirm with kubectl describe pod <pod-name>, measure real usage with kubectl top pod --containers, then either raise the limit to fit the real working set or fix the leak. For JVM and Node apps, also cap the runtime heap below the container limit.

start here

kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Diagnosis

Causes at a glance

Cause	How to recognize it	Fix
Limit set below the real working set	Kills happen at the same usage level, shortly after warm-up	Measure with kubectl top / metrics and raise resources.limits.memory
Memory leak	Usage climbs steadily for hours or days before every kill	Profile the app and fix the leak: a bigger limit only delays the kill
Runtime heap ignores the container limit	JVM or Node allocates past the limit under load	Set MaxRAMPercentage / --max-old-space-size below the limit
Traffic spikes or batch jobs	Kills correlate with load peaks or cron schedules	Size limits for peak usage; move batch work into its own pod
No limits plus node memory pressure	Pod dies during node MemoryPressure; neighbors evicted too	Set explicit requests and limits so behavior is predictable

How OOMKilled works under the hood

Each container runs in a cgroup whose memory cap is set from resources.limits.memory. When the cgroup tries to allocate beyond the cap, the kernel OOM killer terminates a process inside it (usually your main process) and the container exits 137. This is separate from kubelet eviction: OOMKilled is the kernel enforcing a per-container limit instantly; Evicted is the kubelet shedding pods under node-level pressure.

Because the kill is a kernel action, the app gets no signal it can act on. There is no "almost out of memory" callback: headroom and monitoring are the only early warning.

Step 1

Step 1: confirm the kill and measure real usage

Check the last terminated state for the OOMKilled reason, then compare the limit against what the container actually uses. If the pod is restarting repeatedly it will also show CrashLoopBackOff: the OOM kill is the underlying cause.

kubectl describe pod <pod-name> | grep -B 2 -A 6 "Last State"
kubectl top pod <pod-name> --containers

# historical shape (Prometheus):
# container_memory_working_set_bytes{pod="<pod-name>"}

Step 2

Step 2: set requests and limits that match reality

Requests are what the scheduler reserves; limits are the hard cap the kernel enforces. Size the limit for the observed peak working set plus 20-30% headroom. For memory specifically, a widely used practice is requests equal to limits: memory is not compressible, so promising less than the cap invites node pressure and unpredictable evictions.

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "1Gi"

Step 3

Step 3: cap the runtime heap below the container limit

A container-aware runtime still needs configuration. The JVM defaults to a max heap of only 25% of container memory (wasteful) but an explicit -Xmx above the limit is fatal; set -XX:MaxRAMPercentage=75.0 instead. For Node.js, set --max-old-space-size to roughly 75-80% of the limit. The gap is not waste: threads, off-heap buffers, GC overhead, and native libraries live there.

# JVM
JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75.0"

# Node.js with a 2Gi limit
NODE_OPTIONS="--max-old-space-size=1536"

Leak or undersizing? Read the memory curve

The usage graph tells you which problem you have. A step up at startup that plateaus just above the limit means undersizing: raise the limit and move on. A steady upward slope across hours or days that always ends in a kill is a leak: raising the limit just lengthens the sawtooth. Restart-based mitigation (a scheduled rollout) buys time, but the fix is in the application.

OOMKilled vs Evicted

OOMKilled: kernel enforcement of the per-container limit, exit 137, container restarts in place per restartPolicy. Evicted: the kubelet responding to node-level memory pressure by terminating whole pods, starting with those furthest over their requests; the pod is failed and rescheduled rather than restarted in place. Accurate requests protect you from the second; accurate limits determine the first.

Beyond the one-off fix

Right-sizing memory is a cost decision as much as a reliability one, and the same discipline applies to AI workloads, where the runaway resource is tokens instead of bytes. Nexus gives AI-agent fleets the equivalent of requests and limits: per-agent budgets, token cost attribution, and hard policy caps enforced in-flight, so an agent cannot quietly consume 10x its expected spend the way an unbounded container consumes a node.

Explore Swfte Nexus

Common questions

Why is my pod OOMKilled when kubectl top shows usage below the limit?: kubectl top reports periodic samples, and the kill happens at the instant of an allocation spike between samples. Short-lived surges (large request bodies, batch loads, GC churn) can cross the limit without ever appearing in sampled metrics.
Should memory requests equal limits in Kubernetes?: For memory, it is a common and defensible default. Memory cannot be reclaimed like CPU, so scheduling a pod on less than it may use creates node pressure and eviction risk. Requests = limits gives that resource Guaranteed-style predictability.
Does an OOMKilled container restart automatically?: Yes, per the pod restartPolicy (Always for Deployments), and repeated kills put it into CrashLoopBackOff with growing delays. The restart does not fix anything: usage will climb back to the limit unless you change the limit or the app.
How much headroom should I leave between heap size and the container limit?: Around 20-25%. The runtime needs memory beyond the heap: thread stacks, code cache, off-heap and native buffers. A 2Gi container with a 1.5Gi heap is a reasonable starting ratio; tune from observed working-set data.
Can I see what the app was doing when it was OOMKilled?: Not from the kill itself: SIGKILL leaves no application trace. You reconstruct it from metrics and logs leading up to the kill, or configure the runtime ahead of time (e.g. JVM heap dumps, allocation profiling) so the next kill leaves evidence.