EKS nodes is in NotReady state

one of your EKS nodes is in NotReady state and you want to safely drain and terminate it.

I’ll give you the correct production-safe steps.


✅ Step 1: Confirm Node Status

kubectl get nodes -o wide

Check which node is NotReady.

Then inspect it:

kubectl describe node <node-name>

Common reasons:

  • Kubelet stopped
  • Instance networking issue
  • Disk pressure
  • Instance unhealthy in EC2

✅ Step 2: Cordon the Node (Prevent New Pods)

kubectl cordon <node-name>

This marks it unschedulable.


✅ Step 3: Drain the Node Safely

⚠️ Important: This will evict pods and reschedule them to other nodes.

For normal workloads:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

If stuck due to PDB (PodDisruptionBudget), you may need:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force

If pods are stuck terminating:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --grace-period=30 --timeout=2m

✅ Step 4: Delete Node from Kubernetes

After drain completes:

kubectl delete node <node-name>

✅ Step 5: Terminate EC2 Instance

Now go to:

AWS Console → EC2 → Instances

Find the instance ID of that node and:

👉 Terminate instance


🚀 If Node is in Managed Node Group

If you’re using EKS Managed Node Group, better method:

Instead of manual termination:

aws eks update-nodegroup-config \
  --cluster-name <cluster-name> \
  --nodegroup-name <nodegroup-name> \
  --scaling-config minSize=X,maxSize=Y,desiredSize=Z

Or simply terminate the instance — ASG will automatically launch new one.


⚠️ Important Production Checks

Before draining:

  1. Check if workloads have replicas: kubectl get deploy -A
  2. Check PDB: kubectl get pdb -A
  3. Check cluster capacity: kubectl top nodes

Make sure other nodes can handle workload.


🔥 If Node is Completely Dead (Cannot Drain)

If node is not reachable:

kubectl delete node <node-name>

Then terminate instance in EC2.

Leave a Reply

Your email address will not be published. Required fields are marked *