背景介紹
使用 Kubeadm 搭建的集羣默認證書有效期是一年,證書過期後集羣會無法使用,因此必須在證書過期前及時續期證書。
如果集羣證書過期,我們在使用 kubectl 連接集羣時會收到如下報錯:
Unable to connect to the server: x509: certificate has expired or is not yet valid
過期的證書會阻止對 etcd 集羣的訪問,因為實例間無法相互通信,etcd 日誌可能包含如下錯誤條目:
W | rafthttp: health check for peer 6221a1d241bb2d0a could not connect: x509: certificate
has expired or is not yet valid
I | embed: rejected connection from "10.200.0.4:46108" (error "remote error: tls: bad
certificate", ServerName "")
可以通過如下命令檢查集羣證書的有效期: kubeadm certs check-expiration
[root@master-01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 01, 2026 09:46 UTC 286d ca no
apiserver Jul 01, 2026 09:46 UTC 286d ca no
apiserver-etcd-client Jul 01, 2026 09:46 UTC 286d etcd-ca no
apiserver-kubelet-client Jul 01, 2026 09:46 UTC 286d ca no
controller-manager.conf Jul 01, 2026 09:46 UTC 286d ca no
etcd-healthcheck-client Jul 01, 2026 09:46 UTC 286d etcd-ca no
etcd-peer Jul 01, 2026 09:46 UTC 286d etcd-ca no
etcd-server Jul 01, 2026 09:46 UTC 286d etcd-ca no
front-proxy-client Jul 01, 2026 09:46 UTC 286d front-proxy-ca no
scheduler.conf Jul 01, 2026 09:46 UTC 286d ca no
super-admin.conf Jul 01, 2026 09:46 UTC 286d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jun 29, 2035 09:46 UTC 9y no
etcd-ca Jun 29, 2035 09:46 UTC 9y no
front-proxy-ca Jun 29, 2035 09:46 UTC 9y no
這個集羣證書有效期到 2026年7月1日,一般建議在證書到期前一個月左右就得續期證書。
證書更新
環境介紹
這個測試環境有3台master節點,1台worker 節點,已經運行了78天。
[root@master-01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master-01 Ready control-plane 78d v1.32.2
master-02 Ready control-plane 78d v1.32.2
master-03 Ready control-plane 78d v1.32.2
worker-01 Ready <none> 78d v1.32.2
證書備份
在更新證書之前建議先備份 現有證書,對運維人員來説在任何變更前都備份是一個好習慣,關鍵時刻是救命的稻草。使用 Kubeadm 部署的集羣證書都放在 /etc/kubernetes/pki/ 目錄下。
[root@master-01 ~]# ll /etc/kubernetes/pki/
total 56
-rw-r--r-- 1 root root 1306 Jul 1 17:46 apiserver.crt
-rw-r--r-- 1 root root 1123 Jul 1 17:46 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Jul 1 17:46 apiserver-etcd-client.key
-rw------- 1 root root 1679 Jul 1 17:46 apiserver.key
-rw-r--r-- 1 root root 1176 Jul 1 17:46 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Jul 1 17:46 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1107 Jul 1 17:46 ca.crt
-rw------- 1 root root 1675 Jul 1 17:46 ca.key
drwxr-xr-x 2 root root 162 Jul 1 17:46 etcd
-rw-r--r-- 1 root root 1123 Jul 1 17:46 front-proxy-ca.crt
-rw------- 1 root root 1679 Jul 1 17:46 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 Jul 1 17:46 front-proxy-client.crt
-rw------- 1 root root 1679 Jul 1 17:46 front-proxy-client.key
-rw------- 1 root root 1675 Jul 1 17:46 sa.key
-rw------- 1 root root 451 Jul 1 17:46 sa.pub
可以將 /etc/kubernetes 目錄都備份了,kubernetes 相關的大多數配置文件和證書都放在這裏。
cp -r /etc/kubernetes/ /tmp/k8s-backup
更新第一台 Master 節點證書
我有 3 台 Master 節點 ,先在 第一台 Master 節點上執行命令 kubeadm certs renew all 。
[root@master-01 ~]# kubeadm certs renew all
[renew] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[renew] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
沒有報錯證書就更新成功了,重新檢查證書有效期,可以看到證書有效期已經更新到 2026年9月18日。
[root@master-01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Sep 18, 2026 08:31 UTC 364d ca no
apiserver Sep 18, 2026 08:31 UTC 364d ca no
證書更新成功後不會生效,需要重啓控制面組件,控制面組件都是靜態Pod,重啓時可以把清單文件 /etc/kubernetes/manifests/*.yaml 移走,等20秒左右再移回來就實現了重啓。
mkdir -p /tmp/manifests
mv /etc/kubernetes/manifests/*.yaml /tmp/manifests
sleep 20
mv /tmp/manifests/*.yaml /etc/kubernetes/manifests/
更新剩餘 Master 節點
在另外幾台 Master 節點上也執行相同的操作即可,一台更新成功後再更新另一台,Master 節點都配置的負載均衡,對 Worker 節點是沒有影響的,還是建議在業務低峯期操作。
kubeadm certs renew all
mkdir -p /tmp/manifests
mv /etc/kubernetes/manifests/*.yaml /tmp/manifests
sleep 20
mv /tmp/manifests/*.yaml /etc/kubernetes/manifests/
更新 kubeconfig 文件
證書更新成功後還需要更新 kubeconfig 文件 。
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
99 年證書
每年更新證書確實很麻煩,在 kubeadm v1beta4 版本中增加了定義證書有效期的字段,我們可以把證書修改為 99 年,這樣就再也不用為續期證書煩惱了。
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
certificateValidityPeriod: 8760h # 默認:365 天 × 24 小時 = 1 年
caCertificateValidityPeriod: 87600h # 默認:365 天 × 24 小時 * 10 = 10 年
修改為
certificateValidityPeriod: 876000h # 99 年
caCertificateValidityPeriod: 876000h # 99 年
檢查集羣證書,都變成了 99 年,是不是很爽。
[root@master-01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 26, 2125 08:36 UTC 99y ca no
apiserver Aug 26, 2125 08:36 UTC 99y ca no
...
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 26, 2125 08:36 UTC 99y no
etcd-ca Aug 26, 2125 08:36 UTC 99y no
front-proxy-ca Aug 26, 2125 08:36 UTC 99y no
總結
本文演示瞭如何更新 Kubernetes 集羣證書,如果是新搭建集羣就可以通過修改參數將集羣證書修改為99年,這樣就不至於忘記更新證書導致的故障了。