节点重启流程 检查 etcd 状态etcd 推荐奇数节点部署以保证 quorum多数派存活时集群可正常读写。容错计算公式⌊n/2⌋ 1参考官方容错性说明。当前 3 节点需至少保证 2 个 etcd 存活。以下操作需在每台 etcd 节点上验证## 验证数据一致性 / 节点健康 # etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint status --write-outtable ## DB SIZE: 数据库大小部署时通过 --quota-backend-bytes 设置上限默认 2G ## IS LEADER: 是否为 leader ## IS LEARNER: 是否为非投票成员worker ## RAFT TERM: leader 任期须保证各节点该值一致重启 / 网络抖动都会使其 1 -------------------------------------------------------------------------------------------------------------------------------------- | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | -------------------------------------------------------------------------------------------------------------------------------------- | https://x.x.x.x:2379 | 683c58b549788bd9 | 3.5.15 | 30 MB | true | false | 40 | 130863104 | 130863104 | | -------------------------------------------------------------------------------------------------------------------------------------- ## 验证连通性 / 响应延迟 # etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health --write-outtable ## HEALTH: 能否读写 ## TOOK: 读一个随机 key无错误即判定健康耗时 ≈ 网络往返 leader 心跳确认 ---------------------------------------------------- | ENDPOINT | HEALTH | TOOK | ERROR | ---------------------------------------------------- | https://x.x.x.x:2379 | true | 60.842745ms | | ---------------------------------------------------- ## 查看成员列表 # etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ member list -w table ## STATUS: 节点状态 ## PEER ADDRS: 节点间通信地址 ## CLIENT ADDRS: 客户端API server访问地址 ## IS LEARNER: 是否为非投票成员worker ---------------------------------------------------------------------------------- | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | ---------------------------------------------------------------------------------- | xxxxxxx | started | demo-1 | https://x.x.x.x:2380 | https://x.x.x.x:2379 | false | | xxxxxxx | started | demo-2 | https://x.x.x.x:2380 | https://x.x.x.x:2379 | false | | xxxxxxx | started | demo-3 | https://x.x.x.x:2380 | https://x.x.x.x:2379 | false | ----------------------------------------------------------------------------------2.阻止 Pod 调度通过cordon标记待重启节点阻止新 Pod 调度上来注一次只操作一台节点完成该节点的 cordon → 重启 → uncordon 流程后才能处理下一台# kubectl cordon demo-13.驱逐 Podkube-apiserver/controller-manager/scheduler/etcd是 kubelet 直接管理的静态 Podstatic poddrain 不会驱逐它们。只要其余两个节点的 etcd 存活集群控制面就正常。# kubectl drain demo-1 \ --ignore-daemonsets \ --delete-emptydir-data \ --timeout300s驱逐后确认节点已停止调度# kubectl get node NAME STATUS ROLES AGE VERSION demo-1 Ready,SchedulingDisabled control-plane 336d v1.31.14 demo-2 Ready control-plane 336d v1.31.14 demo-3 Ready control-plane 336d v1.31.14常见报错驱逐超时通常是因为 PDBPodDisruptionBudget不允许驱逐例如error when evicting pods/prometheus-k8s-0 -n monitoring: Cannot evict pod as it would violate the pods disruption budget.排查并处理该节点的 PDB除临时改 PDB 外也可扩容对应副本数或手动清理 Pod但这会使 PDB 失去意义。# kubectl get pdb -n monitoring prometheus-k8s NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE prometheus-k8s 1 N/A 0 337d ## 临时将 minAvailable 调为 0结束后建议还原 # kubectl patch pdb prometheus-k8s -n monitoring --typejson -p[{op:replace,path:/spec/minAvailable,value:0}]4.重启节点# ssh demo-1 # reboot5.恢复调度并验证节点重启后确认状态恢复Ready内核版本也会变为更新后的版本再用uncordon解除调度限制# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME demo-1 Ready,SchedulingDisabled control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://1.7.27 demo-2 Ready control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-64-generic containerd://1.7.27 demo-3 Ready control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-106-generic containerd://1.7.27 # kubectl uncordon demo-1 node/demo-1 uncordoned # kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME demo-1 Ready control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://1.7.27 demo-2 Ready control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-64-generic containerd://1.7.27 demo-3 Ready control-plane 336d v1.31.14 x.x.x.x none Ubuntu 24.04.4 LTS 6.8.0-106-generic containerd://1.7.276.循环操作当前节点处理完毕回到第 1 步对下一台节点重复整个流程...问题排查1.重启后 Pod 被标记为invalid1.1.问题现象重启节点后该节点所有 Pod 的 RESTARTS 列显示为invalid静态 Pod READY 列展示为 0/1。在 k8s 源码中pod 创建时间与当前时间偏差超过 2 秒即显示 invalid。# kubectl get pods -A -o wide | grep demo-3 NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system coredns-dbbb9ff68-z8wjd 1/1 Running 0 (invalid ago) 131m x.x.x.x demo-3 kube-system etcd-demo-3 0/1 Running 0 (invalid ago) 10m x.x.x.x demo-3 kube-system kube-apiserver-demo-3 0/1 Running 0 (invalid ago) 10m x.x.x.x demo-3 kube-system kube-controller-manager-demo-3 0/1 Running 0 (invalid ago) 10m x.x.x.x demo-3 kube-system kube-proxy-f2fj8 1/1 Running 1 (invalid ago) 75d x.x.x.x demo-3 kube-system kube-scheduler-demo-3 0/1 Running 0 (invalid ago) 10m x.x.x.x demo-3 kube-system node-local-dns-xkfxc 1/1 Running 3 (invalid ago) 75d x.x.x.x demo-31.2.排查过程1.2.1.校验容器与节点时间以 etcd Pod 为例其startedAtUTC比节点当前 UTC 时间晚了约 7 小时 40 分处于 未来 时间通过时间结尾的 Z 判断时间格式为 UTC## 节点当前时间CST / UTC # date Wed Jun 17 09:06:16 PM CST 2026 # date -u Wed Jun 17 01:06:16 PM UTC 2026 ## Pod 容器状态时间为 UTC # kubectl get pod -n kube-system etcd-demo-1 -o jsonpath{.status.containerStatuses} | jq . [ { ... lastState: { terminated: { exitCode: 255, finishedAt: 2026-06-17T20:46:05Z, reason: Unknown, startedAt: 2026-06-17T09:56:07Z } }, name: etcd, ready: false, restartCount: 4, state: { running: { startedAt: 2026-06-17T20:46:16Z } } } ] ## containerd 记录的容器创建/启动时间 ## https://github.com/kubernetes-sigs/cri-tools/blob/v1.26.1/cmd/crictl/container.go#L862 # crictl inspect 2ea3cdadfec6f | grep -Ei createdAt|startedAt|finishedAt createdAt: 2026-06-18T04:46:16.1787087808:00, startedAt: 2026-06-18T04:46:16.64195002208:00, finishedAt: 0001-01-01T00:00:00Z, ## 换算为 UTC 统一对比 ## containerd: 2026-06-17 20:46:16 UTC ## 节点当前: 2026-06-17 13:06:16 UTC ← 容器时间在「未来」晚 7h40m1.2.2.校验 Pod 底层容器状态排除 containerd / etcd 异常。底层容器实际处于 Running 状态# crictl ps -a | grep etcd CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD 2ea3cdadfec6f 2e96e5913fc06 Less than a second ago Running etcd 4 7451b51061d22 etcd-demo-1 fab531bce16e7 2e96e5913fc06 3 hours ago Exited etcd 3 5f897d72fd206 etcd-demo-1 ## 进程确实在跑 # ps aux | grep -v grep | grep etcd root 2089 ... etcd --advertise-client-urlshttps://x.x.x.x:2379 ... ## etcd 节点状态 # etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint status -w table -------------------------------------------------------------------------------------------------------------------------------------- | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | -------------------------------------------------------------------------------------------------------------------------------------- | https://x.x.x.x:2379 | 683c58b549788bd9 | 3.5.15 | 30 MB | false | false | 59 | 131325454 | 131325454 | | --------------------------------------------------------------------------------------------------------------------------------------1.2.3.定位时间问题根因各节点都配了 NTP对比时间一致。这里我的想法是这样的containerd 服务本身已经很多年了大概率不会有这种 bug更可能是他创建容器时识别到的时间就是在 未来...那现在查看时间却正常了应该是被节点中配置的时间同步改回来了。所以我才通过 dmesg 看一下启动后的内核记录