one of your EKS nodes is in NotReady state and you want to safely drain and terminate it.
I’ll give you the correct production-safe steps.
✅ Step 1: Confirm Node Status
kubectl get nodes -o wide
Check which node is NotReady.
Then inspect it:
kubectl describe node <node-name>
Common reasons:
- Kubelet stopped
- Instance networking issue
- Disk pressure
- Instance unhealthy in EC2
✅ Step 2: Cordon the Node (Prevent New Pods)
kubectl cordon <node-name>
This marks it unschedulable.
✅ Step 3: Drain the Node Safely
⚠️ Important: This will evict pods and reschedule them to other nodes.
For normal workloads:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
If stuck due to PDB (PodDisruptionBudget), you may need:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force
If pods are stuck terminating:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --grace-period=30 --timeout=2m
✅ Step 4: Delete Node from Kubernetes
After drain completes:
kubectl delete node <node-name>
✅ Step 5: Terminate EC2 Instance
Now go to:
AWS Console → EC2 → Instances
Find the instance ID of that node and:
👉 Terminate instance
🚀 If Node is in Managed Node Group
If you’re using EKS Managed Node Group, better method:
Instead of manual termination:
aws eks update-nodegroup-config \
--cluster-name <cluster-name> \
--nodegroup-name <nodegroup-name> \
--scaling-config minSize=X,maxSize=Y,desiredSize=Z
Or simply terminate the instance — ASG will automatically launch new one.
⚠️ Important Production Checks
Before draining:
- Check if workloads have replicas:
kubectl get deploy -A - Check PDB:
kubectl get pdb -A - Check cluster capacity:
kubectl top nodes
Make sure other nodes can handle workload.
🔥 If Node is Completely Dead (Cannot Drain)
If node is not reachable:
kubectl delete node <node-name>
Then terminate instance in EC2.
Leave a Reply