2020-04-20 13:37:31.471260 W | rafthttp: health check for peer 4898f4a4483db9e3 could not connect: dial tcp 172.17.21.60:2380: getsockopt: connection refused 2020-04-20 13:37:32.273336 I | raft: 923b8cf0a7704360 is starting a new election at term 33044 2020-04-20 13:37:32.273355 I | raft: 923b8cf0a7704360 became candidate at term 33045 2020-04-20 13:37:32.273363 I | raft: 923b8cf0a7704360 received MsgVoteResp from 923b8cf0a7704360 at term 33045 2020-04-20 13:37:32.273371 I | raft: 923b8cf0a7704360 [logterm: 11084, index: 167884379] sent MsgVote request to 4898f4a4483db9e3 at term 33045 2020-04-20 13:37:33.973358 I | raft: 923b8cf0a7704360 is starting a new election at term 33045 2020-04-20 13:37:33.973388 I | raft: 923b8cf0a7704360 became candidate at term 33046 2020-04-20 13:37:33.973395 I | raft: 923b8cf0a7704360 received MsgVoteResp from 923b8cf0a7704360 at term 33046 2020-04-20 13:37:33.973404 I | raft: 923b8cf0a7704360 [logterm: 11084, index: 167884379] sent MsgVote request to 4898f4a4483db9e3 at term 33046 2020-04-20 13:37:34.420803 E | etcdserver: publish error: etcdserver: request timed out 2020-04-20 13:37:35.573354 I | raft: 923b8cf0a7704360 is starting a new election at term 33046 2020-04-20 13:37:35.573381 I | raft: 923b8cf0a7704360 became candidate at term 33047 2020-04-20 13:37:35.573389 I | raft: 923b8cf0a7704360 received MsgVoteResp from 923b8cf0a7704360 at term 33047 2020-04-20 13:37:35.573397 I | raft: 923b8cf0a7704360 [logterm: 11084, index: 167884379] sent MsgVote request to 4898f4a4483db9e3 at term 33047
通过docker logs查看kube-api-server日志,发现如下证书过期的错误:
E0420 13:49:46.438819 1 authentication.go:65] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid] E0420 13:49:46.444011 1 authentication.go:65] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid] E0420 13:49:46.448161 1 authentication.go:65] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
在通过查看kubelet日志:/var/logs/messages日志也发现了证书过期的错误:
Apr 20 18:37:33 k8s01 systemd: Unit kubelet.service entered failed state. Apr 20 18:37:33 k8s01 systemd: kubelet.service failed. Apr 20 18:37:40 k8s01 dhclient[3397]: XMT: Solicit on eth0, interval 131610ms. Apr 20 18:37:43 k8s01 systemd: kubelet.service holdoff time over, scheduling restart. Apr 20 18:37:43 k8s01 systemd: Started kubelet: The Kubernetes Node Agent. Apr 20 18:37:43 k8s01 systemd: Starting kubelet: The Kubernetes Node Agent... Apr 20 18:37:43 k8s01 kubelet: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config -file/ for more information. Apr 20 18:37:43 k8s01 kubelet: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config -file/ for more information. Apr 20 18:37:43 k8s01 systemd: Started Kubernetes systemd probe. Apr 20 18:37:43 k8s01 systemd: Starting Kubernetes systemd probe. Apr 20 18:37:43 k8s01 kubelet: I0420 18:37:43.785502 18338 server.go:407] Version: v1.13.5 Apr 20 18:37:43 k8s01 kubelet: I0420 18:37:43.785637 18338 plugins.go:103] No cloud provider specified. Apr 20 18:37:43 k8s01 kubelet: E0420 18:37:43.787102 18338 bootstrap.go:209] Part of the existing bootstrap client certificate is expired: 2020-04-19 16:11:32 +0000 UTC Apr 20 18:37:43 k8s01 kubelet: F0420 18:37:43.787209 18338 server.go:261] failed to run Kubelet: unable to load bootstrap kubeconfig: Error loading config file "/etc/kubernetes/bootstrap-kubelet.conf": yaml: line 6: could not find e xpected ':' Apr 20 18:37:43 k8s01 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Available Commands: certs Commands related to handling kubernetes certificates # 证书相关 kubeconfig Kubeconfig file utilities kubelet Commands related to handling the kubelet selfhosting Makes a kubeadm cluster self-hosted
Flags: -h, --helphelpfor alpha
Global Flags: --log-file string If non-empty, use this log file --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. --skip-headers If true, avoid header prefixes in the log messages -v, --v Level number for the log level verbosity
Additional help topics: kubeadm alpha phase Invoke subsets of kubeadm functions separately for a manual install
Use "kubeadm alpha [command] --help"for more information about a command.
# kubeadm alpha certs -h Commands related to handling kubernetes certificates
Usage: kubeadm alpha certs [command]
Aliases: certs, certificates
Available Commands: renew Renews certificates for a Kubernetes cluster # 这里就是更新证书
Flags: -h, --helphelpfor certs
Global Flags: --log-file string If non-empty, use this log file --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. --skip-headers If true, avoid header prefixes in the log messages -v, --v Level number for the log level verbosity
Use "kubeadm alpha certs [command] --help"for more information about a command.
# 7个已过期的,正好是这7个。 # kubeadm alpha certs renew -h This command is not meant to be run on its own. See list of available subcommands.
Available Commands: all renew all available certificates apiserver Generates the certificate for serving the Kubernetes API apiserver-etcd-client Generates the client apiserver uses to access etcd apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet etcd-healthcheck-client Generates the client certificate for liveness probes to healtcheck etcd etcd-peer Generates the credentials for etcd nodes to communicate with each other etcd-server Generates the certificate for serving etcd front-proxy-client Generates the client for the front proxy
Flags: -h, --helphelpfor renew
Global Flags: --log-file string If non-empty, use this log file --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. --skip-headers If true, avoid header prefixes in the log messages -v, --v Level number for the log level verbosity
Use "kubeadm alpha certs renew [command] --help"for more information about a command.
备份
先别急,操作之前一定要备份,以防误操作。
mkdir etc.kubernet.bakcup cp -a /etc/kubernetes etc.kubernet.bakcup/
# 主要是告诉kubeadm中Kubernetes集群的版本,以防止它去网上查找,因为被墙,所以会报错如下: # could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"
开始更新:
$ kubeadm alpha certs renew all --config=kubernetes/kubeadm-init.yaml
Available Commands: # 正好四个 admin Generates a kubeconfig file for the admin to use and for kubeadm itself all Generates all kubeconfig files controller-manager Generates a kubeconfig file for the controller manager to use kubelet Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes scheduler Generates a kubeconfig file for the scheduler to use
Flags: -h, --helphelpfor kubeconfig
Global Flags: --log-file string If non-empty, use this log file --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. --skip-headers If true, avoid header prefixes in the log messages -v, --v Level number for the log level verbosity
Use "kubeadm init phase kubeconfig [command] --help"for more information about a command.