Kubernetes

Remove a Control Plane from etcd After Deleting the Node

If you delete a control plane node and receive this error on re-creating it:

error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://10.10.0.3:2379 with maintenance client: context deadline exceeded

You need to remove the old node from etcd. To do this, connect to another control plane and do this:

List all nodes, and copy the ID of the deleted node

$ kubectl exec node01 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
36f7f01d71a2114, started, node01, https://10.10.0.1:2380, https://10.10.0.1:2379, false
22fa48691f987ab9, started, node02, https://10.10.0.2:2380, https://10.10.0.2:2379, false
4bad0a237bfeb2c6, started, node03, https://10.10.0.3:2380, https://10.10.0.3:2379, false

In this case, the old node is node03, with ID 4bad0a237bfeb2c6.

Remove it

$ kubectl exec etcd-prd-sms-dc03-z2-kubernetes-controlplane04 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove 4bad0a237bfeb2c6
Member 4bad0a237bfeb2c6 removed from cluster d64f0c95b8a1f61d