It is very convenient to use kubeadm to install kubernetes cluster, but there is also a more annoying problem is that the default certificate is only valid for one year, so you need to consider the issue of certificate upgrade, the demo cluster version of this article is v1.16.2 version, there is no guarantee that the following operation is also applicable to other versions, before the operation must first backup the certificate directory, to prevent the operation error rollback . Make sure to backup the certificate directory before operation to prevent rollback of operation errors . This article introduces two ways to update the cluster certificate.

Manual certificate renewal

The client certificate generated by kubeadm is only valid for one year by default, and we can check if the certificate is expired by using the check-expiration command.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 07, 2020 11:59 UTC   73d             no
apiserver                  Nov 07, 2020 11:59 UTC   73d             no
apiserver-etcd-client      Nov 07, 2020 11:59 UTC   73d             no
apiserver-kubelet-client   Nov 07, 2020 11:59 UTC   73d             no
controller-manager.conf    Nov 07, 2020 11:59 UTC   73d             no
etcd-healthcheck-client    Nov 07, 2020 11:59 UTC   73d             no
etcd-peer                  Nov 07, 2020 11:59 UTC   73d             no
etcd-server                Nov 07, 2020 11:59 UTC   73d             no
front-proxy-client         Nov 07, 2020 11:59 UTC   73d             no
scheduler.conf             Nov 07, 2020 11:59 UTC   73d             no

This command displays the expiration/remaining time of client certificates in the /etc/kubernetes/pki folder and the client certificates embedded in the KUBECONFIG file used by kubeadm.

kubeadm cannot manage certificates signed by external CAs, so if you have an external certificate, you need to manage the certificate renewal manually.

Also note that kubelet.conf is not included in the above list, because kubeadm configures kubelet to automatically renew certificates.

In addition, kubeadm automatically updates all certificates when the control panel is upgraded, so it is best practice to upgrade your cluster frequently with kubeadm to ensure that your cluster stays up to date and reasonably secure. However, for real production environments we may not upgrade the cluster as often as we would like, so we need to update the certificates manually.

To renew certificates manually is also very easy, we just need to renew your certificates with the kubeadm alpha certs renew command, which performs the renewal with the CA (or front-proxy-CA) certificate and the key stored in /etc/kubernetes/pki.

If you are running a highly available cluster, this command needs to be executed on all control panel nodes.

Next, let’s update our cluster certificates. The following actions are performed on the master node, starting with a backup of the original certificates.

1
2
3

$ mkdir /etc/kubernetes.bak
$ cp -r /etc/kubernetes/pki/ /etc/kubernetes.bak
$ cp /etc/kubernetes/*.conf /etc/kubernetes.bak

Then back up the etcd data directory.

1

$ cp -r /var/lib/etcd /var/lib/etcd.bak

Next, execute the command to renew the certificate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ kubeadm alpha certs renew all --config=kubeadm.yaml
kubeadm alpha certs renew all --config=kubeadm.yaml
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Through the above command certificate on a key to update completed, this time to view the above certificate can see the expiration time is already a year after the time of.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Aug 26, 2021 03:47 UTC   364d            no
apiserver                  Aug 26, 2021 03:47 UTC   364d            no
apiserver-etcd-client      Aug 26, 2021 03:47 UTC   364d            no
apiserver-kubelet-client   Aug 26, 2021 03:47 UTC   364d            no
controller-manager.conf    Aug 26, 2021 03:47 UTC   364d            no
etcd-healthcheck-client    Aug 26, 2021 03:47 UTC   364d            no
etcd-peer                  Aug 26, 2021 03:47 UTC   364d            no
etcd-server                Aug 26, 2021 03:47 UTC   364d            no
front-proxy-client         Aug 26, 2021 03:47 UTC   364d            no
scheduler.conf             Aug 26, 2021 03:47 UTC   364d            no

Then remember to update the kubeconfig file.

1
2
3
4
5
6

$ kubeadm init phase kubeconfig all --config kubeadm.yaml
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"

Overwrite the original admin file with the newly generated admin configuration file:

1
2
3

$ mv $HOME/.kube/config $HOME/.kube/config.old
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ chown $(id -u):$(id -g) $HOME/.kube/config

After restarting kube-apiserver, kube-controller, kube-scheduler, and etcd, we can check the validity of the apiserver certificate to verify that the update was successful.

1
2

$ echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate
notAfter=Aug 26 03:47:23 2021 GMT

You can see that the expiration date is now one year past, proving that the update has been successful.

Renewing Certificates with the Kubernetes Certificate API

In addition to the above one-click manual certificate update, you can also use the Kubernetes Certificate API to perform manual certificate updates. For online environments we may not take the risk of updating the cluster or renewing the certificate often, which is risky after all, so we want to generate a certificate with a long enough validity. There are many administrators who manually change the kubeadm source code to 10 years and then recompile it to create a cluster. This approach is not recommended, although it can serve the purpose, especially if you want to update the cluster with a new version. In fact, Kubernetes provides an API to help us generate a long enough certificate to be valid.

To sign using the built-in API method, first we need to configure the --experimental-cluster-signing-duration parameter of the kube-controller-manager component, adjusting it to 10 years:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ vi /etc/kubernetes/manifests/kube-controller-manager.yaml
......
spec:
  containers:
  - command:
    - kube-controller-manager
    # 设置证书有效期为 10 年
    - --experimental-cluster-signing-duration=87600h 
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
......

After the changes are made kube-controller-manager will automatically restart to take effect. Then we need to create a certificate signing request for the Kubernetes certificate API using the following command. If you set up an external signer such as cert-manager, certificate signing requests (CSRs) will be automatically approved. Otherwise, you must manually approve the certificate using the kubectl certificate command. The following kubeadm command outputs the name of the certificate to be approved and then waits for the approval to occur.

1

$ kubeadm alpha certs renew all --use-api --config kubeadm.yaml &

The output is similar to the following.

1
2

[1] 2890
[certs] Certificate request "kubeadm-cert-kubernetes-admin-pn99f" created

Then next we need to go to manually approve the certificate:.

1
2
3
4
5
6

$ kubectl get csr
NAME                                  AGE   REQUESTOR          CONDITION
kubeadm-cert-kubernetes-admin-pn99f   64s   kubernetes-admin   Pending
# 手动批准证书
$ kubectl certificate approve kubeadm-cert-kubernetes-admin-pn99f
certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kubernetes-admin-pn99f approved

Approve the csr in the Pending state in the same way until all csr’s are approved. The final status of the list of all csr’s is shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ kubectl get csr
NAME                                                AGE     REQUESTOR          CONDITION
kubeadm-cert-front-proxy-client-llhrj               30s     kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-2s6kf                   2m43s   kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-etcd-client-t9pkx       2m7s    kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-kubelet-client-pjbjm    108s    kubernetes-admin   Approved,Issued
kubeadm-cert-kube-etcd-healthcheck-client-8dcn8     64s     kubernetes-admin   Approved,Issued
kubeadm-cert-kubernetes-admin-pn99f                 4m29s   kubernetes-admin   Approved,Issued
kubeadm-cert-system:kube-controller-manager-mr86h   79s     kubernetes-admin   Approved,Issued
kubeadm-cert-system:kube-scheduler-t8lnw            17s     kubernetes-admin   Approved,Issued
kubeadm-cert-ydzs-master-cqh4s                      52s     kubernetes-admin   Approved,Issued
kubeadm-cert-ydzs-master-lvbr5                      41s     kubernetes-admin   Approved,Issued

The validity of the inspection certificate after the completion of the approval.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 05, 2029 11:53 UTC   9y              no
apiserver                  Nov 05, 2029 11:54 UTC   9y              no
apiserver-etcd-client      Nov 05, 2029 11:53 UTC   9y              no
apiserver-kubelet-client   Nov 05, 2029 11:54 UTC   9y              no
controller-manager.conf    Nov 05, 2029 11:54 UTC   9y              no
etcd-healthcheck-client    Nov 05, 2029 11:53 UTC   9y              no
etcd-peer                  Nov 05, 2029 11:53 UTC   9y              no
etcd-server                Nov 05, 2029 11:54 UTC   9y              no
front-proxy-client         Nov 05, 2029 11:54 UTC   9y              no
scheduler.conf             Nov 05, 2029 11:53 UTC   9y              no

We can see that it has been extended by 10 years, because the ca certificate is only valid for 10 years.

However, we can’t restart several components of the control panel directly yet, because the etcd corresponding to the cluster installed with kubeadm uses the /etc/kubernetes/pki/etcd/ca.crt certificate by default, and the certificate approved with the command kubectl certificate approve is issued with the default /etc/kubernetes/pki/ca.crt certificate, so we need to replace the ca authority certificate in etcd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# 先拷贝静态 Pod 资源清单
$ cp -r /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak
$ vi /etc/kubernetes/manifests/etcd.yaml
......
spec:
  containers:
  - command:
    - etcd
    # 修改为 CA 文件
    - --peer-trusted-ca-file=/etc/kubernetes/pki/ca.crt
    - --trusted-ca-file=/etc/kubernetes/pki/ca.crt
......
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki  # 更改证书目录
      name: etcd-certs
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki  # 将 pki 目录挂载到 etcd 中去
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd 
      type: DirectoryOrCreate
    name: etcd-data
......

Since kube-apiserver has to connect to the etcd cluster, the corresponding etcd ca file also needs to be reworked:

1
2
3
4
5
6
7
8
9

$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
......
spec:
  containers:
  - command:
    - kube-apiserver
    # 将etcd ca文件修改为默认的ca.crt文件
    - --etcd-cafile=/etc/kubernetes/pki/ca.crt
......

In addition, you need to replace the requestheader-client-ca-file file, which by default is the /etc/kubernetes/pki/front-proxy-ca.crt file, with the default CA file as well, otherwise using the aggregation API, for example, after installing the metrics- server and then execute the kubectl top command, it will report the following error.

1
2

$ cp /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/front-proxy-ca.crt
$ cp /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/front-proxy-ca.key

Since it is a static Pod, the above components will be restarted automatically after the changes are made. Since our current version of kubelet has certificate auto-rotation enabled by default, I don’t need to manage the kubelet certificate anymore, so I update the certificate to 10 validity. Make sure to backup the certificate directory before operation to prevent rollback of operation errors .