What happens to pods on a NotReady node?

They keep running on the node (if it is alive) but the control plane marks them for eviction: after the default node.kubernetes.io/not-ready toleration of 300 seconds, controllers reschedule replacements elsewhere. StatefulSet pods with volumes may need the node confirmed dead before they can move.

What is the difference between Ready=False and Ready=Unknown?

False means the kubelet is alive and reporting a specific problem: read the condition reason. Unknown means the kubelet has stopped reporting entirely: the process is dead, the node is down, or the network path to the API server is broken.

How do I see kubelet logs?

On the node, run journalctl -u kubelet -f (it runs as a systemd service on virtually all distributions). On managed clusters without node SSH, use the cloud provider's node log export or serial console.

Does restarting the kubelet affect running containers?

No: containers are managed by the container runtime (containerd) and keep running while the kubelet restarts. A kubelet restart is a low-risk first move for a wedged node.

Why does my node flap between Ready and NotReady?

Flapping usually means the kubelet is alive but struggling to renew its heartbeat in time: an overloaded node (CPU starvation of system daemons), intermittent network to the API server, or an overloaded control plane. Check node load and kubelet logs for PLEG or heartbeat errors.

Error reference

Kubernetes NodeNotReady: Diagnosis and Recovery

Updated July 27, 2026

In short

NodeNotReady means the node's kubelet is either reporting an unhealthy condition (status False) or has stopped reporting at all (status Unknown: no heartbeat for about 40 seconds). The usual suspects: a dead or wedged kubelet, disk or memory pressure, a broken container runtime or CNI, or the VM itself being gone. Start with kubectl describe node <node-name> and read the Conditions block: it states which check failed and why. Pods on the node are evicted after the default 5-minute not-ready toleration.

start here

kubectl describe node <node-name> | grep -A 10 Conditions

Diagnosis

Causes at a glance

Cause	How to recognize it	Fix
kubelet down or wedged	Ready condition is Unknown; "Kubelet stopped posting node status"	SSH in: systemctl status kubelet, read journalctl -u kubelet, restart it
Disk pressure	DiskPressure condition True; df shows the disk nearly full	Prune unused images and logs; grow the disk; check kubelet GC thresholds
Memory pressure	MemoryPressure condition True; system OOM events on the node	Evict/reschedule heavy pods; fix requests so nodes are not overpacked
Network or CNI failure	NetworkUnavailable True, or "container runtime network not ready"	Check the CNI daemonset pods on that node and node connectivity
VM terminated or unreachable	No SSH, cloud console shows instance stopped/preempted	Replace the node; rely on cluster-autoscaler or the node group to heal

What Ready actually means

The kubelet continuously reports node conditions and renews a heartbeat lease. Ready=True means the kubelet is healthy and able to accept pods. Ready=False means the kubelet itself reports a problem. Ready=Unknown means the control plane has heard nothing for the node-monitor grace period (about 40 seconds by default): the kubelet or the node is unreachable.

False and Unknown point in different directions: False comes with a reason from the kubelet you can act on; Unknown means you are debugging the kubelet process, the node, or the network path to the API server.

Step 1

Step 1: read the node conditions and events

The Conditions block names the failing check (MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable, Ready) with a reason and message. Events on the node object add the timeline.

kubectl get nodes
kubectl describe node <node-name>
# Conditions:
#   MemoryPressure  False
#   DiskPressure    True   KubeletHasDiskPressure
#   Ready           False  ...

Step 2

Step 2: check the kubelet on the node

For Unknown status, get onto the node and look at the kubelet directly. Crash loops in the kubelet log usually name the cause: certificate expiry, a full disk preventing writes, a container runtime (containerd) that is down, or misconfiguration after an upgrade.

systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager
# container runtime healthy?
systemctl status containerd
crictl ps

Step 3

Step 3: clear disk or memory pressure

DiskPressure commonly comes from image bloat and logs. The kubelet garbage-collects images when disk usage passes its high threshold (85% by default), but it cannot delete images in use or non-image data. Free space manually, then confirm the condition clears.

df -h /var/lib/containerd /var/log
crictl rmi --prune          # remove unused images
journalctl --vacuum-size=500M

Step 4

Step 4: check networking and the CNI

If the runtime reports "network plugin not ready" or NetworkUnavailable is True, the CNI agent on that node (calico-node, aws-node, cilium, kube-proxy) is the suspect. Check whether its daemonset pod on this node is running, and whether the node can reach the API server at all.

kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=<node-name>
# from the node:
curl -k https://<api-server>:6443/healthz

Step 5

Step 5: recover the workloads, then the node

Pods on a not-ready node are evicted automatically after the default not-ready toleration (300 seconds), but you can move faster: cordon the node so nothing new lands there, drain it to reschedule workloads, and then fix or replace it. In managed node groups, deleting the bad instance is often quicker than repairing it.

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
# after fixing:
kubectl uncordon <node-name>

Beyond the one-off fix

Node failures trigger automation: autoscalers replace instances, runbooks drain nodes, and increasingly AI agents execute those runbooks. A cordon or drain issued by an agent looks identical to one issued by an SRE unless something records the difference. Nexus captures every agent-run kubectl action with its full context, so when half a node pool was drained at 3 a.m., you know whether it was self-healing working as designed or an agent that needs a tighter policy.

Explore Swfte Nexus

Common questions

What happens to pods on a NotReady node?: They keep running on the node (if it is alive) but the control plane marks them for eviction: after the default node.kubernetes.io/not-ready toleration of 300 seconds, controllers reschedule replacements elsewhere. StatefulSet pods with volumes may need the node confirmed dead before they can move.
What is the difference between Ready=False and Ready=Unknown?: False means the kubelet is alive and reporting a specific problem: read the condition reason. Unknown means the kubelet has stopped reporting entirely: the process is dead, the node is down, or the network path to the API server is broken.
How do I see kubelet logs?: On the node, run journalctl -u kubelet -f (it runs as a systemd service on virtually all distributions). On managed clusters without node SSH, use the cloud provider's node log export or serial console.
Does restarting the kubelet affect running containers?: No: containers are managed by the container runtime (containerd) and keep running while the kubelet restarts. A kubelet restart is a low-risk first move for a wedged node.
Why does my node flap between Ready and NotReady?: Flapping usually means the kubelet is alive but struggling to renew its heartbeat in time: an overloaded node (CPU starvation of system daemons), intermittent network to the API server, or an overloaded control plane. Check node load and kubelet logs for PLEG or heartbeat errors.