Yesterday there was a problem with the network environment, the local virtual machine built Kubernetes environment does not have a fixed IP, the result of the node IP changed, of course the easiest way is to re-fix the node back to the previous IP address, but I stubbornly want to modify the IP address of the cluster, the results encountered a lot of problems, and not as simple as I thought.

Environment

First look at the previous environment.

1
2
3
4

➜  ~ cat /etc/hosts
192.168.0.111 master1
192.168.0.109 node1
192.168.0.110 node2

New IP address.

1
2
3
4

➜  ~ cat /etc/hosts
192.168.0.106 master1
192.168.0.101 node1
192.168.0.105 node2

So we need to change the IP addresses of all nodes.

Operation steps

First change the /etc/hosts of all nodes to the new addresses.

Before manipulating any files it is highly recommended to backup .

master node

  1. Back up the /etc/kubernetes directory.

    1

    ➜ cp -Rf /etc/kubernetes/ /etc/kubernetes-bak

  2. Replace the APIServer address of all configuration files in /etc/kubernetes.

    1
2
3
4
5
6
7
8

    ➜ oldip=192.168.0.111
➜ newip=192.168.0.106
# 查看之前的
➜ find . -type f | xargs grep $oldip
# 替换IP地址
➜ find . -type f | xargs sed -i "s/$oldip/$newip/"
# 检查更新后的
➜ find . -type f | xargs grep $newip

  3. Identify the certificate in /etc/kubernetes/pki that has the old IP address as the alt name.

    1
2
3
4
5
6

    ➜ cd /etc/kubernetes/pki
➜ for f in $(find -name "*.crt"); do
openssl x509 -in $f -text -noout > $f.txt;
done
➜ grep -Rl $oldip .
➜ for f in $(find -name "*.crt"); do rm $f.txt; done

  4. Find the ConfigMap in the kube-system namespace that references the old IP.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    # 获取所有的 kube-system 命名空间下面所有的 ConfigMap
➜ configmaps=$(kubectl -n kube-system get cm -o name | \
awk '{print $1}' | \
cut -d '/' -f 2)

# 获取所有的ConfigMap资源清单
➜ dir=$(mktemp -d)
➜ for cf in $configmaps; do
kubectl -n kube-system get cm $cf -o yaml > $dir/$cf.yaml
done

# 找到所有包含旧 IP 的 ConfigMap
➜ grep -Hn $dir/* -e $oldip

# 然后编辑这些 ConfigMap，将旧 IP 替换成新的 IP
➜ kubectl -n kube-system edit cm kubeadm-config
➜ kubectl -n kube-system edit cm kube-proxy

    This step is very, very important. I neglected this step when I was working on it, which caused Flannel CNI to fail to start and keep reporting errors, similar to the log message below.

    1
2
3
4

    ➜ kubectl logs -f kube-flannel-ds-pspzf -n kube-system
I0512 14:46:26.044229       1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[ens33] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0512 14:46:26.044617       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0512 14:46:56.142921       1 main.go:222] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-pspzf': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-pspzf": dial tcp 10.96.0.1:443: i/o timeout

    Actually, I couldn’t connect to the apiserver, and it took me a long time to check the kube-proxy logs, which showed the following error message.

    1

    E0512 14:53:03.260817       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.0.111:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.0.111:6443: connect: no route to host

    This is because the apiserver address configured in kube-proxy’s ConfigMap is the old IP address, so be sure to replace it with the new one.

  5. Delete the certificate and private key grep’d in step 3 and regenerate them.

    1
2
3
4
5
6

    ➜ cd /etc/kubernetes/pki
➜ rm apiserver.crt apiserver.key
➜ kubeadm init phase certs apiserver

➜ rm etcd/peer.crt etcd/peer.key
➜ kubeadm init phase certs etcd-peer

    It is of course possible to regenerate all of them.

    1

    ➜ kubeadm init phase certs all

  6. Generate a new kubeconfig file.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    ➜ cd /etc/kubernetes
➜ rm -f admin.conf kubelet.conf controller-manager.conf scheduler.conf
➜ kubeadm init phase kubeconfig all
I0513 15:33:34.404780   52280 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.22
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
# 覆盖默认的 kubeconfig 文件
➜ cp /etc/kubernetes/admin.conf $HOME/.kube/config

  7. Restart the kubelet.

    1
2

    ➜ systemctl restart containerd
➜ systemctl restart kubelet

    The normal Kubernetes clusters are now accessible.

    1
2
3
4
5

    ➜ kubectl get nodes
NAME      STATUS     ROLES                  AGE   VERSION
master1   Ready      control-plane,master   48d   v1.22.8
node1     NotReady   <none>                 48d   v1.22.8
node2     NotReady   <none>                 48d   v1.22.8

node nodes

Although the cluster is now accessible, we can see that the Node node is now in the NotReady state, and we can go check the kubelet logs for the node2 node.

1
2
3
4
5
6

➜ journalctl -u kubelet -f
......
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.470896    1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.531695    1194 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.0.111:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.0.111:6443: connect: no route to host
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.571958    1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.673379    1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"

You can see that the previous APIServer address is still being accessed, so where is the APIServer address being used explicitly? We can check the kubelet startup parameters with the following command.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

➜ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2022-05-13 14:37:31 CST; 1h 13min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 1194 (kubelet)
    Tasks: 15
   Memory: 126.9M
   CGroup: /system.slice/kubelet.service
           └─1194 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kub...

May 13 15:51:08 node2 kubelet[1194]: E0513 15:51:08.787677    1194 kubelet.go:2412] "Error getting node" err="node \"node2... found"
May 13 15:51:08 node2 kubelet[1194]: E0513 15:51:08.888194    1194 kubelet.go:2412] "Error getting node" err="node \"node2... found"
......

The core configuration file is /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf and its contents are shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

➜ cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

There is a configuration KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet. conf, where two configuration files bootstrap-kubelet.conf and kubelet.conf are mentioned, the first of which does not exist.

1
2

➜ cat /etc/kubernetes/bootstrap-kubelet.conf
cat: /etc/kubernetes/bootstrap-kubelet.conf: No such file or directory

The second configuration file is in the form of a kubeconfig file, and this file specifies the address of the APIServer, which you can see is still the same IP address as before.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

➜ cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <......>
    server: https://192.168.0.111:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

So our first thought must be to change the APIServer address here to a new IP address, but this is obviously problematic because the relevant certificate is still the same as before and needs to be regenerated, so how do we regenerate the file?

The first step is to backup the kubelet working directory.

1
2

➜ cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak
➜ cp -rf /var/lib/kubelet/ /var/lib/kubelet-bak

Remove the kubelet client certificate.

1

➜ rm /var/lib/kubelet/pki/kubelet-client*

Then go generate the kubelet.conf file at the master1 node (the node with the /etc/kubernetes/pki/ca.key file).

1
2

# 在master1节点
➜ kubeadm kubeconfig user --org system:nodes --client-name system:node:node2 --config kubeadm.yaml > kubelet.conf

Then copy the kubelet.conf file to the node2 node /etc/kubernetes/kubelet.conf, then restart the kubelet on the node2 node and wait for /var/lib/kubelet/pki/kubelet-client-current. pem is recreated.

1
2
3
4
5
6
7
8

➜ systemctl restart kubelet
# 重启后等待重新生成 kubelet 客户端证书
➜ ll /var/lib/kubelet/pki/
total 12
-rw------- 1 root root 1106 May 13 16:32 kubelet-client-2022-05-13-16-32-35.pem
lrwxrwxrwx 1 root root   59 May 13 16:32 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2022-05-13-16-32-35.pem
-rw-r--r-- 1 root root 2229 Mar 26 14:39 kubelet.crt
-rw------- 1 root root 1675 Mar 26 14:39 kubelet.key

Ideally we can point to the rotating kubelet client certificate by manually editing kubelet.conf, replacing client-certificate-data and client-key-data in the file with /var/lib/kubelet/pki/kubelet- client-current.pem.

1
2

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

Restart the kubelet again and the node2 node will now be Ready. Just configure the node1 node again in the same way.

The above operation can accomplish our needs normally, but it requires some knowledge of the relevant certificates. There is a simpler way to do this.

First, stop the kubelet and back up the directory you want to work on.

1
2
3

➜ systemctl stop kubelet
➜ mv /etc/kubernetes /etc/kubernetes-bak
➜ mv /var/lib/kubelet/ /var/lib/kubelet-bak

Preserve the pki certificate directory.

1
2
3
4
5
6
7

➜ mkdir -p /etc/kubernetes
➜ cp -r /etc/kubernetes-bak/pki /etc/kubernetes
➜ rm /etc/kubernetes/pki/{apiserver.*,etcd/peer.*}
rm: remove regular file '/etc/kubernetes/pki/apiserver.crt'? y
rm: remove regular file '/etc/kubernetes/pki/apiserver.key'? y
rm: remove regular file '/etc/kubernetes/pki/etcd/peer.crt'? y
rm: remove regular file '/etc/kubernetes/pki/etcd/peer.key'? y

Now we use the following command to reinitialize the control plane nodes, but the most important point is to use etcd’s data directory, which can be done by telling kubeadm to use pre-existing etcd data with the --ignore-preflight-errors=DirAvailable--var-lib-etcd flag.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

➜ kubeadm init --config kubeadm.yaml --ignore-preflight-errors=DirAvailable--var-lib-etcd
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
        [WARNING DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api.k8s.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1] and IPs [10.96.0.1 192.168.0.106]
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master1] and IPs [192.168.0.106 127.0.0.1 ::1]
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 12.003599 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.106:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:27993cae9c76d18a1b82b800182c4c7ebc7a704ba1093400ed886f65e709ec04

The above operation is almost the same as when we go to initialize the cluster, the only difference is the addition of the --ignore-preflight-errors=DirAvailable--var-lib-etcd parameter, which means that the data from the previous etcd is used. Then we can verify that the IP address of the APIServer has changed to the new address.

1
2
3
4
5
6
7

➜ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
➜ kubectl cluster-info
Kubernetes control plane is running at https://192.168.0.106:6443
CoreDNS is running at https://192.168.0.106:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

For node nodes we can reset and rejoin the cluster.

1
2

# 在node节点操作
➜ kubeadm reset

Just reset and rejoin the cluster.

1
2
3

# 在node节点操作
➜ kubeadm join 192.168.0.106:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:27993cae9c76d18a1b82b800182c4c7ebc7a704ba1093400ed886f65e709ec04

This way is much easier than the above way. The cluster is also normal after normal operation.

1
2
3
4
5

➜ kubectl get nodes
NAME      STATUS   ROLES                  AGE     VERSION
master1   Ready    control-plane,master   48d     v1.22.8
node1     Ready    <none>                 48d     v1.22.8
node2     Ready    <none>                 4m50s   v1.22.8

Summary

It is best to use static IP addresses for Kubernetes cluster nodes to avoid the impact of IP changes on the business. If it is not a static IP, it is also highly recommended to add a custom domain name for signing, so that when the IP changes, you can directly remap the domain name. You only need to configure apiServer.certSANs in the kubeadm configuration file via ClusterConfiguration, as shown below.

1
2
3
4
5
6
7
8
9

apiVersion: kubeadm.k8s.io/v1beta3
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
  - api.k8s.local
  - master1
  - 192.168.0.106
kind: ClusterConfiguration
......

Add the address that needs to be signed to the certSANs configuration. For example, here we have added an additional api.k8s.local address so that even if the IP changes in the future you can directly map this domain name to the new IP address, also if you want to access your cluster via an external access IP, then you need to add your external IP address for signature authentication.