It is very convenient to use kubeadm to install kubernetes cluster, but there is also a more annoying problem is that the default certificate is only valid for one year, so you need to consider the issue of certificate upgrade, the demo cluster version of this article is v1.16.2 version, there is no guarantee that the following operation is also applicable to other versions, before the operation must first backup the certificate directory, to prevent the operation error rollback . Make sure to backup the certificate directory before operation to prevent rollback of operation errors . This article introduces two ways to update the cluster certificate.
Manual certificate renewal
The client certificate generated by kubeadm is only valid for one year by default, and we can check if the certificate is expired by using the
This command displays the expiration/remaining time of client certificates in the
/etc/kubernetes/pki folder and the client certificates embedded in the
KUBECONFIG file used by kubeadm.
kubeadmcannot manage certificates signed by external CAs, so if you have an external certificate, you need to manage the certificate renewal manually.
Also note that
kubelet.conf is not included in the above list, because kubeadm configures kubelet to automatically renew certificates.
In addition, kubeadm automatically updates all certificates when the control panel is upgraded, so it is best practice to upgrade your cluster frequently with kubeadm to ensure that your cluster stays up to date and reasonably secure. However, for real production environments we may not upgrade the cluster as often as we would like, so we need to update the certificates manually.
To renew certificates manually is also very easy, we just need to renew your certificates with the
kubeadm alpha certs renew command, which performs the renewal with the CA (or front-proxy-CA) certificate and the key stored in
If you are running a highly available cluster, this command needs to be executed on all control panel nodes.
Next, let’s update our cluster certificates. The following actions are performed on the master node, starting with a backup of the original certificates.
Then back up the etcd data directory.
Next, execute the command to renew the certificate.
Through the above command certificate on a key to update completed, this time to view the above certificate can see the expiration time is already a year after the time of.
Then remember to update the kubeconfig file.
Overwrite the original admin file with the newly generated admin configuration file:
After restarting kube-apiserver, kube-controller, kube-scheduler, and etcd, we can check the validity of the apiserver certificate to verify that the update was successful.
You can see that the expiration date is now one year past, proving that the update has been successful.
Renewing Certificates with the Kubernetes Certificate API
In addition to the above one-click manual certificate update, you can also use the Kubernetes Certificate API to perform manual certificate updates. For online environments we may not take the risk of updating the cluster or renewing the certificate often, which is risky after all, so we want to generate a certificate with a long enough validity. There are many administrators who manually change the kubeadm source code to 10 years and then recompile it to create a cluster. This approach is not recommended, although it can serve the purpose, especially if you want to update the cluster with a new version. In fact, Kubernetes provides an API to help us generate a long enough certificate to be valid.
To sign using the built-in API method, first we need to configure the
--experimental-cluster-signing-duration parameter of the kube-controller-manager component, adjusting it to 10 years:
After the changes are made kube-controller-manager will automatically restart to take effect. Then we need to create a certificate signing request for the Kubernetes certificate API using the following command. If you set up an external signer such as
cert-manager, certificate signing requests (CSRs) will be automatically approved. Otherwise, you must manually approve the certificate using the
kubectl certificate command. The following kubeadm command outputs the name of the certificate to be approved and then waits for the approval to occur.
The output is similar to the following.
Then next we need to go to manually approve the certificate:.
Approve the csr in the Pending state in the same way until all csr’s are approved. The final status of the list of all csr’s is shown below.
The validity of the inspection certificate after the completion of the approval.
We can see that it has been extended by 10 years, because the ca certificate is only valid for 10 years.
However, we can’t restart several components of the control panel directly yet, because the etcd corresponding to the cluster installed with kubeadm uses the
/etc/kubernetes/pki/etcd/ca.crt certificate by default, and the certificate approved with the command
kubectl certificate approve is issued with the default
/etc/kubernetes/pki/ca.crt certificate, so we need to replace the ca authority certificate in etcd:
Since kube-apiserver has to connect to the etcd cluster, the corresponding etcd ca file also needs to be reworked:
In addition, you need to replace the
requestheader-client-ca-file file, which by default is the
/etc/kubernetes/pki/front-proxy-ca.crt file, with the default CA file as well, otherwise using the aggregation API, for example, after installing the metrics- server and then execute the
kubectl top command, it will report the following error.
Since it is a static Pod, the above components will be restarted automatically after the changes are made. Since our current version of kubelet has certificate auto-rotation enabled by default, I don’t need to manage the kubelet certificate anymore, so I update the certificate to 10 validity. Make sure to backup the certificate directory before operation to prevent rollback of operation errors .