Kubernetes 常用运维指令手册

适用版本：Kubernetes 1.35 / 1.36 / 1.37（当前支持窗口，N-2） 配套：containerd 2.x、kubeadm v1beta4、CNI（Calico/Cilium/Flannel）、CSI、k3s 约定：# root 或 sudo，$ 普通用户，<xxx> 为占位符，<ns> 为 namespace。 差异标注：正文用 > **版本差异** 标注跨版本不一致之处。 更新时间：2026-11（版本事实核对至 2026-11-05）

目录

1. 版本与生命周期
2. 架构与组件
3. 集群部署
4. 节点管理
5. kubectl 全解
6. 工作负载
7. 调度
8. 存储
9. 网络
10. 配置与密钥
11. RBAC 与认证
12. 集群安全
13. 集群运维
14. 故障排查
15. 监控与日志
16. 备份与迁移
17. 卸载与重建
18. 高频故障速查
19. 附录

1. 版本与生命周期

1.1 发布模型

Kubernetes 每年 3 个 minor 版本，约 15 周（4 个月） 一发，每个版本维护约 14 个月，只支持最近 3 个 minor（N-2）。

下一版本 1.38 预计 2026-12-16（code freeze 2026-11-17）。

1.2 三个必须知道的时间点

1. cgroup v1 已不可用：1.35 起 kubelet 在 cgroup v1 上默认直接失败。CentOS 7 等老系统不能作为新集群节点。
2. kube-proxy 的 ipvs 模式已弃用（1.35 起）：新集群用 iptables 或 nftables 模式。
3. Ingress NGINX 已退役（2026-03 归档）：新项目直接上 Gateway API，存量集群要规划迁移。

1.3 1.33 → 1.37 重要变更（升级影响）

版本差异：kubeadm.k8s.io/v1beta3 在 1.31 弃用，1.32+ 使用 v1beta4。extraArgs 从 map 改为 [{name, value}] 列表，dryRun/timeout 移出配置文件。抄 1.2x 的旧 yaml 会直接解析失败。

1.4 版本识别

kubectl version -o yaml
kubectl version --short 2>/dev/null || kubectl version
kubectl get nodes -o wide
kubeadm version -o short
kubelet --version
crictl version

# 服务端各组件的版本
kubectl get --raw /version | jq
kubectl -n kube-system get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

1.5 API 废弃检查（升级前必做）

# 1) 官方工具（推荐）：pluto
pluto detect-all-in-cluster
pluto detect-files -d ./manifests

# 2) kubectl 插件
kubectl krew install deprecated-api
kubectl deprecated-api

# 3) 内置：查看 API 资源版本
kubectl api-resources --verbs=list --namespaced -o name | \
  xargs -n1 kubectl get --show-kind --ignore-not-found -A 2>/dev/null | head

# 4) 审计日志找废弃 API 调用
# apiserver 开 audit 后：
grep -o '"resource":"[^"]*"' /var/log/kubernetes/audit.log | sort | uniq -c | sort -rn | head

常见已移除 API（1.29 之后）：

2. 架构与组件

2.1 组件关系

                    ┌────────────────────────────────────────┐
                    │            Control Plane                │
                    │  ┌───────────┐   ┌──────────────────┐  │
   kubectl ────────▶│  │ apiserver │◀──│    etcd          │  │
                    │  └─────┬─────┘   └──────────────────┘  │
                    │        │                                │
                    │  ┌─────▼──────┐  ┌──────────────────┐   │
                    │  │ controller │  │    scheduler     │   │
                    │  │  manager   │  │                  │   │
                    │  └────────────┘  └──────────────────┘   │
                    └────────────┬───────────────────────────┘
                                 │ (watch/list)
        ┌────────────────────────┼────────────────────────┐
        │                        │                        │
   ┌────▼─────┐            ┌─────▼────┐             ┌─────▼────┐
   │ kubelet  │            │ kubelet  │             │ kubelet  │
   │    +     │            │    +     │             │    +     │
   │containerd│            │containerd│             │containerd│
   └────┬─────┘            └────┬─────┘             └────┬─────┘
        │                       │                        │
   ┌────▼─────┐            ┌────▼─────┐             ┌────▼─────┐
   │   Pods   │            │   Pods   │             │   Pods   │
   └──────────┘            └──────────┘             └──────────┘
        └─────────── CNI（跨节点网络）────────────────┘

2.2 控制面高可用基础

• apiserver 无状态 → 多副本 + LB/VIP
• scheduler / controller-manager → 多副本但只有 leader 工作（选举）
• etcd → 需要 奇数个（3/5），基于 Raft，多数派可用

3. 集群部署

3.1 规划

最小生产：3 control-plane + 3 worker，etcd 与 control-plane 混部（≤5 节点可接受）。

网段规划（必须提前定，后期改代价大）：

3.2 所有节点：系统准备

# 主机名
hostnamectl set-hostname k8s-master01

# hosts
cat >>/etc/hosts <<'EOF'
10.0.0.11 k8s-master01
10.0.0.12 k8s-master02
10.0.0.13 k8s-master03
10.0.0.21 k8s-node01
10.0.0.22 k8s-node02
10.0.0.23 k8s-node03
10.0.0.100 api.k8s.local
EOF

# 时间同步（证书与 etcd 强依赖）
dnf install -y chrony && systemctl enable --now chronyd
chronyc sources -v

# 关闭 swap
swapoff -a
sed -ri 's/^([^#].*\sswap\s.*)$/#\1/' /etc/fstab
free -h

# 内核模块与参数
cat >/etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay && modprobe br_netfilter

cat >/etc/sysctl.d/99-k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
# 高并发场景（Service 多时防 conntrack 满）
net.netfilter.nf_conntrack_max      = 1048576
fs.inotify.max_user_watches         = 524288
fs.inotify.max_user_instances       = 1024
EOF
sysctl --system

# 确认 cgroup v2（1.35+ 必须）
stat -fc %T /sys/fs/cgroup/          # 必须输出 cgroup2fs

# SELinux（官方要求 permissive 或 disabled）
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

# 防火墙放行
for p in 6443 2379 2380 10250 10257 10259 8472; do
  firewall-cmd --permanent --add-port=${p}/tcp
done
firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --reload

# 装 iproute-tc（kubeadm 需要）
dnf install -y iproute-tc

坑点：fs.inotify.max_user_watches 太小会导致大量 pod 场景下 kubelet 报 "too many open files"、日志采集异常。生产默认调到 524288。

3.3 所有节点：容器运行时

# 装 containerd（详见 containerd 分册）
dnf install -y containerd.io
containerd config default > /etc/containerd/config.toml

# 两处必改
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
# sandbox_image 与 k8s 版本匹配（1.37 用 pause:3.10）
sed -i 's|sandbox_image = ".*"|sandbox_image = "registry.k8s.io/pause:3.10"|' /etc/containerd/config.toml

systemctl enable --now containerd
systemctl status containerd | head -5

3.4 所有节点：kube 组件

cat >/etc/yum.repos.d/kubernetes.repo <<'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.37/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.37/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

dnf install -y kubelet kubeadm kubectl cri-tools --disableexcludes=kubernetes
systemctl enable kubelet       # 先不 start，等 init 后自动拉起

kubeadm version -o short
kubelet --version

Debian 系：

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
  | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /' \
  > /etc/apt/sources.list.d/kubernetes.list
apt update && apt install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

3.5 镜像准备（国内必需）

kubeadm config images list --kubernetes-version v1.37.2

# 方式一：init 时指定仓库
# --image-repository registry.aliyuncs.com/google_containers

# 方式二：containerd mirror（推荐，全局生效）
mkdir -p /etc/containerd/certs.d/registry.k8s.io
cat >/etc/containerd/certs.d/registry.k8s.io/hosts.toml <<'EOF'
server = "https://registry.k8s.io"
[host."https://registry.aliyuncs.com/google_containers"]
  capabilities = ["pull", "resolve"]
EOF
systemctl restart containerd

# 预拉取
kubeadm config images pull --kubernetes-version v1.37.2
crictl images | grep -E 'kube|pause|etcd'

3.6 单 master 初始化

cat >/tmp/kubeadm-config.yaml <<'EOF'
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.0.11
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  taints: []
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.37.2
clusterName: prod
imageRepository: registry.aliyuncs.com/google_containers
controlPlaneEndpoint: "10.0.0.11:6443"
networking:
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
  dnsDomain: cluster.local
apiServer:
  certSANs:
    - 10.0.0.11
    - api.k8s.local
  timeoutForControlPlane: 4m0s
  extraArgs:
    - name: audit-log-path
      value: /var/log/kubernetes/audit.log
    - name: audit-log-maxage
      value: "30"
    - name: audit-log-maxbackup
      value: "10"
    - name: audit-log-maxsize
      value: "100"
    - name: audit-policy-file
      value: /etc/kubernetes/audit-policy.yaml
  extraVolumes:
    - name: audit-policy
      hostPath: /etc/kubernetes/audit-policy.yaml
      mountPath: /etc/kubernetes/audit-policy.yaml
      readOnly: true
    - name: audit-log
      hostPath: /var/log/kubernetes
      mountPath: /var/log/kubernetes
      pathType: DirectoryOrCreate
controllerManager:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
scheduler:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false
maxPods: 250
serializeImagePulls: false
evictionHard:
  nodefs.available: "10%"
  imagefs.available: "15%"
  memory.available: "500Mi"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables
EOF

# 预检（会检查端口、swap、cgroup、运行时）
kubeadm init --config /tmp/kubeadm-config.yaml --dry-run

# 正式初始化
kubeadm init --config /tmp/kubeadm-config.yaml --upload-certs

初始化成功后：

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl get nodes            # 此时 NotReady，等 CNI
kubectl get pod -n kube-system

3.7 安装 CNI

# Flannel（简单，内网够用，不支持 NetworkPolicy）
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Calico（支持 NetworkPolicy，生产常用）
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.30/manifests/calico.yaml

# 或 Tigera operator
# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30/manifests/tigera-operator.yaml
# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30/manifests/custom-resources.yaml

# Cilium（eBPF，可替代 kube-proxy，性能好）
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.0.0.100 \
  --set k8sServicePort=6443

# 验证
kubectl get pod -n kube-system -o wide | grep -E 'flannel|calico|cilium'
kubectl get nodes

坑点：CNI 的 podCIDR 必须与 kubeadm 的 podSubnet 一致。Flannel 默认 10.244.0.0/16，Calico 默认 192.168.0.0/16——用 Calico 时要改 CALICO_IPV4POOL_CIDR 或 kubeadm 配 192.168.0.0/16。

3.8 加入 worker

# master 上生成
kubeadm token create --print-join-command
# kubeadm join 10.0.0.11:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx

# worker 执行上述命令
# 验证
kubectl get nodes -o wide

3.9 加入 control-plane（HA）

# 在 master01 生成带证书的 join 命令
kubeadm init phase upload-certs --upload-certs
# 输出 certificate key

kubeadm token create --print-join-command --certificate-key <key>

# 在 master02/03 执行
kubeadm join 10.0.0.100:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <key>

3.10 完整 HA（VIP + keepalived）

# 三台 master 上装
dnf install -y keepalived

cat >/etc/keepalived/keepalived.conf <<'EOF'
global_defs {
  router_id k8s-ha
  script_user root
  enable_script_security
}

vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -20
  fall 3
  rise 2
}

vrrp_instance VI_1 {
  state BACKUP
  interface eth0
  virtual_router_id 51
  priority 100
  advert_int 1
  nopreempt
  authentication {
    auth_type PASS
    auth_pass K8sHa123
  }
  virtual_ipaddress {
    10.0.0.100/24
  }
  track_script {
    check_apiserver
  }
}
EOF

# 健康检查脚本
cat >/etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash
err=0
for k in $(seq 1 5); do
  curl -sf --max-time 2 https://127.0.0.1:6443/healthz >/dev/null || err=1
  [ $err -eq 0 ] && break
  sleep 2
done
exit $err
EOF
chmod +x /etc/keepalived/check_apiserver.sh

systemctl enable --now keepalived
ip a | grep 10.0.0.100

# kubeadm 用 VIP 作 controlPlaneEndpoint
# controlPlaneEndpoint: "10.0.0.100:6443"

3.11 k3s（单机/边缘）

# server
curl -sfL https://get.k3s.io | INSTALL_K3S_MIRROR=cn \
  sh -s - server \
  --cluster-init \
  --tls-san 10.0.0.11 \
  --disable traefik \
  --write-kubeconfig-mode 644

cat /var/lib/rancher/k3s/server/node-token

# agent
curl -sfL https://get.k3s.io | INSTALL_K3S_MIRROR=cn \
  K3S_URL=https://10.0.0.11:6443 K3S_TOKEN=<token> sh -

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
k3s kubectl get nodes

# 配置化
cat >/etc/rancher/k3s/config.yaml <<'EOF'
write-kubeconfig-mode: "0644"
tls-san: ["10.0.0.11", "k3s.local"]
disable: ["traefik"]
EOF
systemctl restart k3s

3.12 部署后验收

# 1) 节点
kubectl get nodes -o wide

# 2) 系统组件
kubectl get pod -n kube-system
kubectl get pod -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

# 3) DNS
kubectl run -it --rm dnstest --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
kubectl get svc -n kube-system kube-dns

# 4) 跨节点网络
kubectl create deploy nginx --image=nginx:1.25 --replicas=3
kubectl get pod -o wide
# 进一个 pod ping 另一个 pod IP

# 5) Service
kubectl expose deploy nginx --port=80 --target-port=80
kubectl run -it --rm curltest --image=curlimages/curl --restart=Never -- curl -s nginx

# 6) 证书
kubeadm certs check-expiration

# 7) etcd
kubectl -n kube-system exec etcd-k8s-master01 -- etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

# 8) 清理测试
kubectl delete deploy nginx && kubectl delete svc nginx

4. 节点管理

4.1 查看

kubectl get nodes -o wide
kubectl get nodes --show-labels
kubectl describe node <node>
kubectl top node
kubectl get node <node> -o yaml

# 资源分配情况
kubectl describe node <node> | grep -A8 'Allocated resources'

# 节点上运行的 pod
kubectl get pod -A --field-selector spec.nodeName=<node> -o wide

# 按状态筛选
kubectl get nodes --no-headers | awk '$2!="Ready"{print}'

自定义列：

kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,STATUS:.status.conditions[-1].type,ROLES:.metadata.labels.node-role\.kubernetes\.io/control-plane,VER:.status.nodeInfo.kubeletVersion,IP:.status.addresses[?(@.type=="InternalIP")].address,OS:.status.nodeInfo.osImage'

4.2 标记与调度控制

# cordon：不可调度（新 pod 不进来，已有 pod 不动）
kubectl cordon <node>
kubectl uncordon <node>

# drain：驱逐 pod 并 cordon
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force --timeout=300s
# --force：驱逐非控制器管理的 pod（裸 pod）
# --delete-emptydir-data：删除 emptyDir 数据（会丢临时数据）
# --dry-run=server：先看会做什么

# 恢复
kubectl uncordon <node>

4.3 污点（Taint）

# 语法：key=value:effect
# effect: NoSchedule / PreferNoSchedule / NoExecute

kubectl taint node <node> dedicated=gpu:NoSchedule
kubectl taint node <node> node.kubernetes.io/unschedulable:NoSchedule
kubectl taint node <node> dedicated=gpu:NoSchedule-          # 删除（末尾加 -）
kubectl taint node <node> dedicated-                          # 删该 key 全部

kubectl describe node <node> | grep Taints

# 内置污点
# node.kubernetes.io/not-ready:NoExecute
# node.kubernetes.io/unreachable:NoExecute
# node.kubernetes.io/disk-pressure:NoSchedule
# node.kubernetes.io/memory-pressure:NoSchedule
# node.kubernetes.io/pid-pressure:NoSchedule
# node.kubernetes.io/unschedulable:NoSchedule

4.4 标签（Label）

kubectl label node <node> disktype=ssd
kubectl label node <node> topology.kubernetes.io/zone=az1
kubectl label node <node> node-role.kubernetes.io/worker=""
kubectl label node <node> disktype-                 # 删除
kubectl label node <node> disktype=hdd --overwrite

# 常用标准标签
# topology.kubernetes.io/region
# topology.kubernetes.io/zone
# node.kubernetes.io/instance-type
# kubernetes.io/arch / kubernetes.io/os

4.5 节点扩容与替换

# 扩容：新机器按 3.2-3.4 准备，然后 join
kubeadm join <endpoint> --token <t> --discovery-token-ca-cert-hash sha256:<h>

# 替换：先加新节点，再排空旧节点，最后删除
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <old-node>
# 旧机器上执行
kubeadm reset -f

4.6 节点状态判读

kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'

5. kubectl 全解

5.1 kubeconfig 与多集群

kubectl config view
kubectl config view --minify
kubectl config get-contexts
kubectl config current-context
kubectl config use-context prod
kubectl config set-context --current --namespace=web
kubectl config rename-context old prod

# 合并多个 kubeconfig
KUBECONFIG=~/.kube/a:~/.kube/b kubectl config view --flatten > ~/.kube/merged
export KUBECONFIG=~/.kube/merged

# 提取当前 context 的独立 kubeconfig（给他人用）
kubectl config view --minify --flatten --context=prod > /tmp/prod.kubeconfig

# 用 token 认证
kubectl --server=https://10.0.0.100:6443 --token=<token> --insecure-skip-tls-verify get nodes

5.2 输出格式

# wide
kubectl get pod -o wide
# yaml / json
kubectl get pod web -o yaml
kubectl get pod web -o json | jq '.status'
# name
kubectl get pod -o name
# custom-columns
kubectl get pod -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP'
# jsonpath
kubectl get pod -o jsonpath='{.items[*].metadata.name}'
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'
kubectl get node -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}'
# go-template
kubectl get pod -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
# kyaml（1.37 GA，更易读）
kubectl get pod web -o kyaml
# sort
kubectl get pod --sort-by=.metadata.creationTimestamp
kubectl get pod --sort-by='.status.containerStatuses[0].restartCount'
# 只取某些字段
kubectl get svc -A -o wide
kubectl get events --sort-by=.lastTimestamp

5.3 创建与变更

# 声明式（推荐）
kubectl apply -f app.yaml
kubectl apply -f ./manifests/          # 整个目录
kubectl apply -f https://example.com/app.yaml
kubectl apply -f app.yaml --dry-run=server
kubectl apply -f app.yaml --prune -l app=web      # 删除清单中不存在的资源

# 命令式（快速验证）
kubectl create deploy web --image=nginx:1.25 --replicas=3
kubectl create ns prod
kubectl create configmap cm --from-file=./conf
kubectl create secret generic s --from-literal=k=v
kubectl create job test --image=busybox -- echo hi
kubectl create cronjob backup --image=busybox --schedule="0 2 * * *" -- echo backup
kubectl expose deploy web --port=80 --target-port=80 --type=ClusterIP
kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- sh

# 生成 yaml 不落地
kubectl create deploy web --image=nginx --dry-run=client -o yaml > web.yaml
kubectl run tmp --image=busybox --dry-run=client -o yaml > pod.yaml

# 编辑与打补丁
kubectl edit deploy web
kubectl patch deploy web -p '{"spec":{"replicas":3}}'
kubectl patch deploy web --type=json -p '[{"op":"replace","path":"/spec/replicas","value":3}]'
kubectl set image deploy/web web=nginx:1.26
kubectl set env deploy/web TZ=Asia/Shanghai
kubectl set resources deploy/web --limits=cpu=2,memory=2Gi
kubectl label/annotate ...

5.4 删除

kubectl delete pod web
kubectl delete pod web --grace-period=0 --force        # 卡 Terminating 时
kubectl delete -f app.yaml
kubectl delete deploy web
kubectl delete pod -l app=web
kubectl delete pod --field-selector status.phase=Failed
kubectl delete ns prod                                  # ★ 删 ns 会删其中所有资源
kubectl delete all --all -n test                        # ★ 危险
kubectl delete pvc --all -n test                        # ★ 可能连带删底层存储

# 删除卡住
kubectl patch pod web -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl api-resources --verbs=list --namespaced -o name | \
  xargs -n1 kubectl get -n <ns> --show-kind --ignore-not-found

5.5 等待与监视

kubectl wait --for=condition=Ready pod/web --timeout=120s
kubectl wait --for=condition=Available deploy/web --timeout=300s
kubectl wait --for=delete pod/web --timeout=60s

kubectl get pod -w
kubectl get pod -w -o wide
watch -n2 'kubectl get pod -o wide'
kubectl rollout status deploy/web -w
kubectl logs -f <pod>

5.6 调试与转发

kubectl logs <pod>
kubectl logs <pod> -c <container>
kubectl logs <pod> --previous
kubectl logs <pod> --tail=200 --timestamps
kubectl logs -l app=web --max-log-requests=10
kubectl logs <pod> --since=10m
kubectl logs <pod> --since-time="2026-11-05T08:00:00Z"

kubectl exec -it <pod> -- sh
kubectl exec -it <pod> -c <container> -- bash
kubectl exec <pod> -- env
kubectl exec <pod> -- cat /etc/resolv.conf

kubectl port-forward pod/web 8080:80
kubectl port-forward svc/web 8080:80
kubectl port-forward deploy/web 8080:80
kubectl port-forward svc/web 8080:80 --address 0.0.0.0

kubectl cp <pod>:/app/logs ./logs
kubectl cp ./config.yaml <pod>:/app/config.yaml -c app
kubectl cp <ns>/<pod>:/path ./local

kubectl attach -it <pod>
kubectl debug -it <pod> --image=busybox:1.36 --target=app
kubectl debug <pod> --copy-to=web-debug --image=ubuntu:24.04 --share-processes -it
kubectl debug node/<node> -it --image=ubuntu:24.04

5.7 集群与 API

kubectl cluster-info
kubectl cluster-info dump > /tmp/dump.txt
kubectl get --raw /healthz
kubectl get --raw '/readyz?verbose'
kubectl get --raw /livez
kubectl get --raw /metrics | head -50
kubectl get --raw /api/v1/namespaces
kubectl api-resources
kubectl api-resources --namespaced=false
kubectl api-versions
kubectl explain pod.spec.containers
kubectl explain deploy.spec.strategy --recursive
kubectl auth can-i --list
kubectl auth can-i create pods --as=system:serviceaccount:prod:ci

5.8 效率配置

# 别名
cat >>~/.bashrc <<'EOF'
alias k=kubectl
alias kgp='kubectl get pods -o wide'
alias kgn='kubectl get nodes -o wide'
alias kga='kubectl get all'
alias kd='kubectl describe'
alias kl='kubectl logs -f --tail=200'
alias ke='kubectl exec -it'
alias kaf='kubectl apply -f'
alias kdel='kubectl delete'
export KUBE_EDITOR=vim
EOF
source ~/.bashrc

# 补全
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc

# 插件（krew）
(
  set -x
  cd "$(mktemp -d)" &&
  OS=$(uname | tr '[:upper:]' '[:lower:]') &&
  ARCH=$(uname -m | sed -e 's/x86_64/amd64/' -e 's/aarch64/arm64/') &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-${OS}_${ARCH}.tar.gz" &&
  tar zxvf krew-${OS}_${ARCH}.tar.gz &&
  ./krew-${OS}_${ARCH} install krew
)
kubectl krew install ctx ns neat deprecations tree score
kubectl ctx            # 切换 context
kubectl ns             # 切换 namespace
kubectl tree deploy web

6. 工作负载

6.1 选型

6.2 Deployment 典型模板

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: prod
  labels: { app: web }
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0          # 保证容量不降
      maxSurge: 1
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web, tier: frontend }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: web
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile: { type: RuntimeDefault }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: web }
      containers:
        - name: web
          image: harbor.internal/web/app:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - { name: http, containerPort: 8080, protocol: TCP }
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - name: NODE_NAME
              valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef: { name: db-secret, key: password }
          envFrom:
            - configMapRef: { name: app-config }
          resources:
            requests: { cpu: 500m, memory: 512Mi }
            limits:   { cpu: "2",  memory: 2Gi }
          startupProbe:
            httpGet: { path: /healthz, port: http }
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet: { path: /ready, port: http }
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 3
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 20
            failureThreshold: 3
          lifecycle:
            preStop:
              exec: { command: ["sh","-c","sleep 10 && nginx -s quit"] }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
            - { name: config, mountPath: /app/config, readOnly: true }
      terminationGracePeriodSeconds: 45
      volumes:
        - name: tmp
          emptyDir: { sizeLimit: 1Gi }
        - name: config
          configMap: { name: app-config }

6.3 探针

# 三种检查方式
httpGet: { path: /healthz, port: 8080, httpHeaders: [{name: X-Health, value: "1"}] }
tcpSocket: { port: 6379 }
exec: { command: ["sh","-c","redis-cli ping | grep PONG"] }
grpc: { port: 9090, service: health }        # 1.24+ GA

参数取舍原则

坑点：liveness 探针查下游依赖（如数据库）会导致级联重启——数据库抖动时全部 pod 一起重启，雪崩。liveness 只查自身进程健康，依赖用 readiness。

6.4 滚动更新与回滚

kubectl set image deploy/web web=harbor.internal/web/app:1.0.1 -n prod
kubectl annotate deploy/web kubernetes.io/change-cause="release 1.0.1 by ops" -n prod
kubectl rollout status deploy/web -n prod
kubectl rollout history deploy/web -n prod
kubectl rollout history deploy/web --revision=2 -n prod
kubectl rollout undo deploy/web -n prod
kubectl rollout undo deploy/web --to-revision=1 -n prod
kubectl rollout pause deploy/web
kubectl rollout resume deploy/web
kubectl rollout restart deploy/web

# 金丝雀：两个 Deployment 共享 Service 的 selector
# stable: replicas 9, labels {app: web, track: stable}
# canary: replicas 1, labels {app: web, track: canary}

6.5 StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: prod
spec:
  serviceName: mysql-headless
  replicas: 3
  podManagementPolicy: OrderedReady     # 或 Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate: { partition: 0 }
  selector:
    matchLabels: { app: mysql }
  template:
    metadata:
      labels: { app: mysql }
    spec:
      containers:
        - name: mysql
          image: mysql:8.4
          ports:
            - { name: mysql, containerPort: 3306 }
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom: { secretKeyRef: { name: mysql-secret, key: root } }
          volumeMounts:
            - { name: data, mountPath: /var/lib/mysql }
          resources:
            requests: { cpu: "2", memory: 4Gi }
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests: { storage: 100Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
  namespace: prod
spec:
  clusterIP: None
  selector: { app: mysql }
  ports:
    - { port: 3306, targetPort: 3306, name: mysql }

关键点：每个 pod 有稳定 DNS mysql-0.mysql-headless.prod.svc.cluster.local；扩缩容按序号；删除 StatefulSet 不删 PVC（数据保留）。

kubectl get sts -n prod
kubectl scale sts mysql --replicas=5 -n prod
kubectl rollout status sts/mysql -n prod
kubectl rollout undo sts/mysql -n prod
kubectl delete sts mysql -n prod --cascade=orphan    # 保留 pod

6.6 DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  namespace: kube-system
spec:
  selector:
    matchLabels: { app: node-agent }
  updateStrategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 10% }
  template:
    metadata:
      labels: { app: node-agent }
    spec:
      hostNetwork: true
      hostPID: true
      tolerations:
        - { operator: Exists }            # 容忍所有污点，保证每节点都跑
      containers:
        - name: agent
          image: harbor.internal/ops/node-agent:1.0
          securityContext:
            privileged: true
          volumeMounts:
            - { name: root, mountPath: /host, readOnly: true }
            - { name: cgroup, mountPath: /sys/fs/cgroup, readOnly: true }
      volumes:
        - { name: root, hostPath: { path: / } }
        - { name: cgroup, hostPath: { path: /sys/fs/cgroup } }

6.7 Job / CronJob

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  namespace: prod
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  completions: 1
  parallelism: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: harbor.internal/web/api:1.0.0
          command: ["java","-jar","app.jar","--migrate-only"]
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
  namespace: prod
spec:
  schedule: "0 2 * * *"
  timeZone: Asia/Shanghai
  concurrencyPolicy: Forbid          # Allow / Forbid / Replace
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  suspend: false
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: harbor.internal/ops/backup:1.0

kubectl create job --from=cronjob/backup backup-manual -n prod
kubectl get jobs -n prod
kubectl logs job/migrate -n prod
kubectl delete job migrate -n prod
kubectl patch cronjob backup -p '{"spec":{"suspend":true}}' -n prod

版本差异：spec.timeZone 在 1.27 GA。老版本只能靠容器内 TZ 环境变量。

6.8 扩缩容

# 手动
kubectl scale deploy web --replicas=5 -n prod
kubectl scale sts mysql --replicas=3 -n prod
kubectl scale deploy web --replicas=0 -n prod        # 停服保配置

# HPA（需要 metrics-server）
kubectl autoscale deploy web --min=2 --max=10 --cpu-percent=70 -n prod
kubectl get hpa -n prod
kubectl describe hpa web -n prod

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - { type: Percent, value: 50, periodSeconds: 60 }
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - { type: Pods, value: 4, periodSeconds: 60 }
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }
    - type: Resource
      resource:
        name: memory
        target: { type: Utilization, averageUtilization: 80 }

6.9 PDB（中断预算）

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: prod
spec:
  minAvailable: 2          # 或 maxUnavailable: 1，二选一
  selector:
    matchLabels: { app: web }

kubectl get pdb -n prod
kubectl describe pdb web-pdb -n prod
# drain 时会遵守 PDB，导致 drain 卡住（说明预算不足）

坑点：minAvailable 等于副本数时，drain 会永远卡住（没有可用驱逐额度）。要么留 1 个余量，要么临时调大副本。

7. 调度

7.1 调度流程

Pod 创建 → 调度队列 → 过滤（Filter，可用节点）→ 打分（Score）→ 绑定（Bind）→ kubelet 拉起

Pending 的 pod 一定是卡在 Filter 阶段，事件里会写明原因（Insufficient cpu、node(s) had untolerated taint、node affinity mismatch）。

7.2 nodeSelector（最简单）

spec:
  nodeSelector:
    disktype: ssd

7.3 节点亲和性

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:      # 硬性
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["az1", "az2"]
              - key: node-type
                operator: NotIn
                values: ["gpu"]
      preferredDuringSchedulingIgnoredDuringExecution:     # 软性
        - weight: 80
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]

7.4 Pod 亲和与反亲和

spec:
  affinity:
    podAntiAffinity:
      # 硬性：同 app 的 pod 不能同节点（高可用）
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels: { app: web }
          topologyKey: kubernetes.io/hostname
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels: { app: cache }
            topologyKey: kubernetes.io/hostname

坑点：硬性反亲和 + 副本数 > 节点数 = 部分 pod 永远 Pending。生产优先用 topologySpreadConstraints（更灵活，可 ScheduleAnyway）。

7.5 拓扑分布约束

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule     # 或 ScheduleAnyway
      labelSelector:
        matchLabels: { app: web }
      minDomains: 2

# 查看节点的拓扑标签
kubectl get nodes -L topology.kubernetes.io/zone -L topology.kubernetes.io/region

7.6 污点与容忍

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - operator: Exists                    # 容忍一切

7.7 资源请求与限制

resources:
  requests: { cpu: 500m, memory: 512Mi }
  limits:   { cpu: "2",  memory: 2Gi }

kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.qosClass}{"\n"}{end}'

7.8 优先级与抢占

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "核心业务"

spec:
  priorityClassName: high-priority

7.9 调度排障

# 看 pod 为什么 Pending
kubectl describe pod <pod> | sed -n '/Events:/,$p'

# 看 scheduler 日志
kubectl logs -n kube-system -l component=kube-scheduler --tail=100

# 看节点剩余可分配
kubectl describe node <node> | grep -A10 'Allocated resources'
kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.cpu}{"\t"}{.status.allocatable.memory}{"\n"}{end}'

# 模拟调度（dry-run 不落盘）
kubectl create -f pod.yaml --dry-run=server

8. 存储

8.1 概念关系

StorageClass（供给模板）
     ↓ 动态供给
PV（集群级存储资源） ←──绑定──→ PVC（命名空间级申请）
                                    ↓ 被
                                  Pod 挂载

8.2 常用 StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com          # 按实际 CSI 改
parameters:
  type: gp3
  fsType: ext4
reclaimPolicy: Delete                  # Delete / Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer   # 或 Immediate
mountOptions:
  - noatime

8.3 PVC 使用

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: prod
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

# Pod 中使用
spec:
  containers:
    - name: app
      volumeMounts:
        - { name: data, mountPath: /app/data }
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: app-data

8.4 常用运维命令

kubectl get sc
kubectl get pv
kubectl get pvc -A
kubectl get pvc -n prod -o wide
kubectl describe pvc app-data -n prod
kubectl get pv <pv> -o yaml

# 扩容（SC 需 allowVolumeExpansion）
kubectl patch pvc app-data -n prod -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
kubectl get pvc app-data -n prod -w

# 快照（需 VolumeSnapshotClass）
kubectl get volumesnapshotclass
kubectl get volumesnapshot -A

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: app-data-snap
  namespace: prod
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: app-data

8.5 PVC Pending 排查

kubectl describe pvc app-data -n prod
# 看 Events：waiting for a volume to be created / no persistent volumes available

# 1) SC 是否存在
kubectl get sc
# 2) provisioner 是否正常
kubectl get pod -A | grep -E 'csi|provisioner'
kubectl logs -n kube-system <csi-pod>
# 3) 容量/accessMode 是否可满足
kubectl get pv -o wide
# 4) WaitForFirstConsumer 时无 pod 使用则不创建（正常现象）

8.6 Retain 卷的回收

# SC 为 Retain 时，PVC 删除后 PV 变 Released，数据仍在
kubectl get pv | grep Released

# 复用：清掉 claimRef 后重新绑定
kubectl patch pv <pv> -p '{"spec":{"claimRef":null}}'
kubectl get pv <pv>

# 彻底删除（底层存储要单独删）
kubectl delete pv <pv>

9. 网络

9.1 四层模型

9.2 Service 类型

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: prod
spec:
  type: ClusterIP
  selector: { app: web }
  ports:
    - { name: http, port: 80, targetPort: http, protocol: TCP }
  # sessionAffinity: ClientIP
  # sessionAffinityConfig: { clientIP: { timeoutSeconds: 10800 } }

kubectl get svc -n prod
kubectl get ep -n prod                    # Endpoints，看是否为空
kubectl get endpointslices -n prod
kubectl describe svc web -n prod

关键：Endpoints 为空是 Service 不通的第一原因，通常是 selector 与 pod label 不匹配，或 pod 未 Ready。

9.3 Ingress 与 Gateway API

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: prod
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["web.example.com"]
      secretName: web-tls
  rules:
    - host: web.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port: { number: 80 }

# Gateway API（Ingress 的继任者）
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main
  namespace: infra
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs: [{ name: wildcard-tls }]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
  namespace: prod
spec:
  parentRefs:
    - { name: main, namespace: infra }
  hostnames: ["web.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - { name: web, port: 80 }

版本差异：Ingress NGINX 已于 2026-03 归档退役。存量可用但无安全更新，新项目用 Gateway API（Cilium/Envoy Gateway/Contour 等实现）。

9.4 NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-policy
  namespace: prod
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }
      ports:
        - { protocol: TCP, port: 8080 }
  egress:
    - to:
        - podSelector:
            matchLabels: { app: db }
      ports:
        - { protocol: TCP, port: 3306 }
    - to:                              # DNS 必须放行
        - namespaceSelector: {}
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

坑点：一旦某个 pod 被 NetworkPolicy 选中且 policyTypes 含 Egress，默认拒绝所有出向流量，包括 DNS。必须显式放行 53 端口，否则全是"域名解析失败"。

9.5 DNS 与 CoreDNS

kubectl get pod -n kube-system -l k8s-app=kube-dns
kubectl get svc -n kube-system kube-dns
kubectl get cm -n kube-system coredns -o yaml
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# DNS 测试
kubectl run -it --rm dnstest --image=busybox:1.36 --restart=Never -- sh
# nslookup kubernetes.default
# nslookup web.prod.svc.cluster.local
# nslookup web            # 同 namespace 短名
# nslookup web.prod       # 跨 namespace

DNS 命名规则：

<service>.<namespace>.svc.cluster.local
<pod-ip-dashed>.<namespace>.pod.cluster.local
<statefulset-pod>.<headless-svc>.<namespace>.svc.cluster.local

常见问题：

# 调 ndots 加速解析
spec:
  dnsConfig:
    options:
      - { name: ndots, value: "2" }

9.6 网络排障

# 1) pod 是否拿到 IP
kubectl get pod -o wide

# 2) pod 间连通
kubectl exec -it <pod-a> -- ping -c2 <pod-b-ip>

# 3) Service 是否有 endpoints
kubectl get ep <svc> -n <ns>
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>

# 4) Service 访问
kubectl run -it --rm tmp --image=curlimages/curl --restart=Never -- \
  curl -v http://web.prod.svc.cluster.local

# 5) 节点侧规则
iptables -t nat -L KUBE-SERVICES -n | head
nft list ruleset | grep -c kube

# 6) CNI 状态
kubectl get pod -n kube-system -o wide | grep -E 'calico|cilium|flannel'
kubectl logs -n kube-system <cni-pod> --tail=100

# 7) 抓包（在 pod 所在节点）
kubectl debug node/<node> -it --image=nicolaka/netshoot -- \
  tcpdump -i any -nn host <pod-ip>

10. 配置与密钥

10.1 ConfigMap

kubectl create cm app-config --from-file=./conf -n prod
kubectl create cm app-config --from-literal=LOG_LEVEL=info -n prod
kubectl create cm app-config --from-env-file=./app.env -n prod

kubectl get cm -n prod
kubectl get cm app-config -n prod -o yaml
kubectl describe cm app-config -n prod
kubectl edit cm app-config -n prod
kubectl delete cm app-config -n prod

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: prod
data:
  LOG_LEVEL: info
  app.yaml: |
    server:
      port: 8080
    db:
      host: mysql.prod.svc.cluster.local

使用的三种方式

# 1) 单个键 → 环境变量
env:
  - name: LOG_LEVEL
    valueFrom: { configMapKeyRef: { name: app-config, key: LOG_LEVEL } }

# 2) 全部键 → 环境变量
envFrom:
  - configMapRef: { name: app-config }
  - configMapRef: { name: app-config, optional: true }

# 3) 挂载为文件
volumes:
  - name: config
    configMap:
      name: app-config
      items:
        - { key: app.yaml, path: application.yaml }
      defaultMode: 0644

关键区别：挂载为 volume 的 ConfigMap 更新后会在 1~2 分钟内自动同步（kubelet 周期性刷新）；作为 env/envFrom 引入的不会变，必须重启 pod。

10.2 Secret

kubectl create secret generic db-secret \
  --from-literal=username=app --from-literal=password='S3cr3t' -n prod

kubectl create secret tls web-tls --cert=web.crt --key=web.key -n prod

kubectl create secret docker-registry harbor-cred \
  --docker-server=harbor.internal --docker-username='robot$ci' \
  --docker-password='xxx' -n prod

kubectl get secret -n prod
kubectl get secret db-secret -n prod -o jsonpath='{.data.password}' | base64 -d
kubectl describe secret db-secret -n prod        # 只显示大小，不显示值

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
  namespace: prod
type: Opaque
stringData:                       # 直接写明文，apply 时自动 base64
  username: app
  password: S3cr3t
---
# 不可变 Secret（1.21 GA，性能更好但不允许修改）
apiVersion: v1
kind: Secret
metadata:
  name: fixed-secret
immutable: true
data:
  key: dmFsdWU=

10.3 Secret 的安全现实

Secret 默认只是 base64 编码，不是加密：

# 任何有 get secret 权限的人都能读
kubectl auth can-i get secrets -n prod --as=system:serviceaccount:prod:default

# 缓解手段
# 1) 开启 etcd 静态加密（见 12.4）
# 2) 收紧 RBAC，禁止 list secrets
# 3) 用外部密钥管理（Vault / External Secrets Operator / CSI Secrets Store）
# 4) 不要提交到 Git（用 SealedSecrets / SOPS）

10.4 环境变量 vs 挂载 vs 启动参数

# 引用的字段（Downward API）
env:
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: CPU_LIMIT
    valueFrom: { resourceFieldRef: { containerName: app, resource: limits.cpu } }
volumes:
  - name: podinfo
    downwardAPI:
      items:
        - { path: labels, fieldRef: { fieldPath: metadata.labels } }

11. RBAC 与认证

11.1 模型

User/Group/ServiceAccount ──绑定──▶ Role/ClusterRole ──授予──▶ 权限（verbs + resources）

11.2 ServiceAccount

kubectl create sa web -n prod
kubectl get sa -n prod
kubectl describe sa web -n prod

# 1.24+ 不再自动创建长期 token secret，需要时手工签发
kubectl create token web -n prod --duration=24h
kubectl create token web -n prod --duration=8760h > /tmp/token

apiVersion: v1
kind: ServiceAccount
metadata:
  name: web
  namespace: prod
automountServiceAccountToken: false      # 不需要 API 访问时关掉（安全）

11.3 Role / RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: prod
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: web-pod-reader
  namespace: prod
subjects:
  - kind: ServiceAccount
    name: web
    namespace: prod
  - kind: User
    name: alice
    apiGroup: rbac.authorization.k8s.io
  - kind: Group
    name: dev-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

11.4 ClusterRole 示例

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes", "nodes/status"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["nodes", "pods"]
    verbs: ["get", "list"]

11.5 权限排查

kubectl auth can-i create pods -n prod
kubectl auth can-i --list -n prod
kubectl auth can-i --list --as=system:serviceaccount:prod:web -n prod
kubectl auth can-i get secrets --as=alice -n prod
kubectl auth whoami

# 谁能做什么
kubectl get rolebinding,clusterrolebinding -A -o wide | grep <user>

# 用 API 查
kubectl get clusterrolebinding -o json | jq -r '
  .items[] | select(.subjects[]?.name=="alice") | .roleRef.name'

11.6 认证方式

# OIDC 配置（apiserver）
# --oidc-issuer-url=https://keycloak.example.com/realms/k8s
# --oidc-client-id=kubernetes
# --oidc-username-claim=email
# --oidc-groups-claim=groups

11.7 最小权限原则

# 给 CI 的 SA：只能更新指定 deployment 的镜像
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deploy-updater
  namespace: prod
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["web", "api"]        # 限定具体资源
    verbs: ["get", "patch", "update"]

禁止给业务 SA 绑 cluster-admin，也不要给 secrets 的 list 权限（list 会返回所有 secret 的完整内容）。

12. 集群安全

12.1 Pod Security Admission（PSA）

1.25 起取代 PSP（PodSecurityPolicy 已移除）。按命名空间打标签：

kubectl label ns prod pod-security.kubernetes.io/enforce=restricted
kubectl label ns prod pod-security.kubernetes.io/enforce-version=v1.37
kubectl label ns prod pod-security.kubernetes.io/audit=baseline
kubectl label ns prod pod-security.kubernetes.io/warn=restricted

kubectl get ns prod --show-labels

坑点：restricted 会拒绝没设 seccompProfile 和 runAsNonRoot 的 pod。存量工作负载要先在 audit/warn 模式跑一段时间，看清楚影响再 enforce。

# 检查哪些 pod 会违规（warn 模式的输出在 kubectl apply 时可见）
kubectl label ns prod pod-security.kubernetes.io/warn=restricted --overwrite
kubectl apply -f app.yaml    # 会输出 Warning: would violate PodSecurity

12.2 seccomp / AppArmor

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault        # RuntimeDefault / Localhost / Unconfined
  containers:
    - name: app
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]
          add: ["NET_BIND_SERVICE"]

# AppArmor（1.30+ 用 securityContext.appArmorProfile）
securityContext:
  appArmorProfile:
    type: RuntimeDefault

12.3 准入控制

# LimitRange：给 ns 内 pod 设默认资源
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: prod
spec:
  limits:
    - type: Container
      default: { cpu: 500m, memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 128Mi }
      max: { cpu: "2", memory: 2Gi }

# ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: prod
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"

kubectl get resourcequota -n prod
kubectl describe resourcequota prod-quota -n prod
kubectl get limitrange -n prod

12.4 etcd 静态加密

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets", "configmaps"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64 32 字节>
      - identity: {}

head -c 32 /dev/urandom | base64

# apiserver 加参数
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# 加密已有数据（重要：新数据才加密，旧数据要重写）
kubectl get secrets -A -o json | kubectl replace -f -

# 验证
kubectl get secret db-secret -n prod -o jsonpath='{.data.password}' | base64 -d   # 仍能读
# etcd 里已经是密文
ETCDCTL_API=3 etcdctl get /registry/secrets/prod/db-secret --print-value-only | head -c 100

12.5 审计日志

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources: [{ group: "", resources: ["endpoints", "services"] }]
  - level: Metadata
    resources: [{ group: "", resources: ["secrets", "configmaps"] }]
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources: [{ group: "", resources: ["pods", "services"] }]
  - level: RequestResponse
    users: ["system:admin"]
  - level: Metadata
    omitStages: ["RequestReceived"]

12.6 安全检查清单

# 1) 是否有 cluster-admin 泛滥
kubectl get clusterrolebinding -o json | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .subjects[]?.name'

# 2) 是否有特权 pod
kubectl get pod -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.privileged==true) | .metadata.namespace+"/"+.metadata.name'

# 3) hostPath 挂载
kubectl get pod -A -o json | jq -r '.items[] | select(.spec.volumes[]?.hostPath) | .metadata.namespace+"/"+.metadata.name'

# 4) 默认 SA 是否 automount
kubectl get sa -A -o json | jq -r '.items[] | select(.automountServiceAccountToken!=false) | .metadata.namespace+"/"+.metadata.name'

# 5) 证书有效期
kubeadm certs check-expiration

# 6) 组件版本与 CVE
kubectl version
kubectl get nodes -o wide

# 7) 镜像是否来自内部仓库
kubectl get pod -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u

# 8) 网络策略覆盖
kubectl get networkpolicy -A

13. 集群运维

13.1 证书管理

# 查看所有证书到期时间
kubeadm certs check-expiration

# 续期全部（kubeadm 集群默认 1 年）
kubeadm certs renew all

# 续期单项
kubeadm certs renew apiserver
kubeadm certs renew apiserver-etcd-client
kubeadm certs renew etcd-healthcheck-client

# 生效：控制面静态 Pod 需重启
# 方式一：删 pod（kubelet 会重建）
kubectl -n kube-system delete pod -l tier=control-plane
# 方式二：移走 manifest（更稳）
mkdir -p /tmp/manifests-bak && mv /etc/kubernetes/manifests/*.yaml /tmp/manifests-bak/
sleep 20
mv /tmp/manifests-bak/*.yaml /etc/kubernetes/manifests/

# kubelet 客户端证书（kubeadm 默认自动轮换）
# /var/lib/kubelet/pki/kubelet-client-current.pem
ls -l /var/lib/kubelet/pki/

# 非 kubeadm 集群：手工用 CA 重签，注意 SAN 必须包含所有 endpoint
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A2 'Subject Alternative Name'

必做：把证书到期检查加进监控。kubeadm 集群默认一年，忘记续期会在某天凌晨全站挂掉。

13.2 etcd 运维

# 定义别名（控制面节点）
cat >>~/.bashrc <<'EOF'
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key'
EOF
source ~/.bashrc

etcdctl endpoint health
etcdctl endpoint status -w table
etcdctl member list -w table
etcdctl alarm list

备份（每天必做）

#!/usr/bin/env bash
# /opt/scripts/etcd-backup.sh
set -euo pipefail
BACKUP_DIR=/backup/etcd
mkdir -p "$BACKUP_DIR"
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save "$BACKUP_DIR/etcd-$(date +%F-%H%M).db"

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot status "$BACKUP_DIR/etcd-$(date +%F-%H%M).db" -w table

# 保留 7 天
find "$BACKUP_DIR" -name 'etcd-*.db' -mtime +7 -delete

chmod +x /opt/scripts/etcd-backup.sh
echo '0 2 * * * /opt/scripts/etcd-backup.sh >> /var/log/etcd-backup.log 2>&1' | crontab -

恢复（灾难场景，务必演练）

# 1) 停控制面
systemctl stop kubelet
mv /etc/kubernetes/manifests /tmp/manifests-bak

# 2) 确认 etcd 容器已停
crictl ps | grep etcd

# 3) 恢复
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/etcd-2026-11-05-0200.db \
  --data-dir=/var/lib/etcd-restore \
  --name=k8s-master01 \
  --initial-cluster=k8s-master01=https://10.0.0.11:2380 \
  --initial-advertise-peer-urls=https://10.0.0.11:2380

# 4) 改 etcd 静态 Pod 的数据目录
# /etc/kubernetes/manifests/etcd.yaml 里 hostPath 改为 /var/lib/etcd-restore

# 5) 起服务
mv /tmp/manifests-bak /etc/kubernetes/manifests
systemctl start kubelet

# 6) 验证
kubectl get nodes
kubectl get pod -A

etcd 性能问题

etcdctl check perf                      # 综合性能
etcdctl defrag --cluster                # 碎片整理（会阻塞，逐节点做）
etcdctl compact $(etcdctl endpoint status --write-out=json | jq '.[0].Status.header.revision')
etcdctl alarm disarm

# 常见指标
# etcd_server_leader_changes_seen_total  → 频繁选主，网络或磁盘问题
# etcd_disk_wal_fsync_duration_seconds   → > 10ms 说明磁盘慢
# etcd_server_proposals_committed_total  → 提案积压

铁律：etcd 必须用 SSD。HDD 上 fsync 延迟高会导致 apiserver 超时、集群"假死"（kubectl 卡住但 pod 还在跑）。

13.3 升级

升级路径：只能一个 minor 一个 minor 升，1.35 → 1.36 → 1.37，不能跳。

# 0) 前置检查
kubeadm upgrade plan
kubeadm certs check-expiration
kubectl get nodes
# 所有节点 Ready 且版本一致；etcd 已备份；已读 release notes 的废弃项

# 1) 升级第一个控制面
dnf install -y kubeadm-1.37.2 --disableexcludes=kubernetes
kubeadm upgrade apply v1.37.2 -y

# 2) 升级 kubelet/kubectl（同一节点）
dnf install -y kubelet-1.37.2 kubectl-1.37.2 --disableexcludes=kubernetes
systemctl daemon-reload
systemctl restart kubelet

# 3) 其余控制面
kubectl drain <cp-node> --ignore-daemonsets --delete-emptydir-data
dnf install -y kubeadm-1.37.2 --disableexcludes=kubernetes
kubeadm upgrade node
dnf install -y kubelet-1.37.2 kubectl-1.37.2 --disableexcludes=kubernetes
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon <cp-node>

# 4) worker 逐台
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
dnf install -y kubeadm-1.37.2 kubelet-1.37.2 kubectl-1.37.2 --disableexcludes=kubernetes
kubeadm upgrade node
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon <node>

# 5) 验证
kubectl get nodes -o wide
kubectl get pod -A | grep -v Running
kubectl version

升级回滚：Kubernetes 不支持降级。唯一可靠的回滚是 etcd 快照恢复 + 重装旧版本二进制。所以升级前必须备份 etcd。

13.4 集群扩容

# 加 worker：新机器按 3.2-3.4 准备
kubeadm join <endpoint>:6443 --token <t> --discovery-token-ca-cert-hash sha256:<h>

# token 过期（默认 24h）时重新生成
kubeadm token create --print-join-command
kubeadm token list

# 加 control-plane
kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command --certificate-key <key>

13.5 节点维护标准流程

# 1) 标记不可调度
kubectl cordon <node>

# 2) 驱逐（注意 PDB，可能卡住）
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=600s

# 3) 维护操作（升级内核、换硬件、扩磁盘）

# 4) 恢复
kubectl uncordon <node>

# 5) 验证
kubectl get node <node>
kubectl get pod -A --field-selector spec.nodeName=<node>

drain 卡住的三种原因：① PDB 预算不足；② 有裸 pod（无控制器，需 --force）；③ DaemonSet pod 未加 --ignore-daemonsets。

13.6 控制面故障处理

# 控制面组件是静态 Pod，看容器状态
crictl ps -a | grep -E 'kube-apiserver|kube-controller|kube-scheduler|etcd'
crictl logs <container-id> --tail 100
journalctl -u kubelet -n 100 | grep -iE 'apiserver|etcd'

14. 故障排查

14.1 分层排查法

1. 集群层：kubectl get --raw /readyz?verbose
2. 节点层：kubectl get nodes / describe node
3. 工作负载层：kubectl get pod / describe pod / logs
4. 容器层：crictl ps / crictl logs / exec
5. 应用层：日志、探针、依赖

# 一键看集群整体
kubectl get --raw='/readyz?verbose'
kubectl get nodes
kubectl get pod -A --field-selector status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -30

14.2 节点 NotReady

# 1) 节点状态与原因
kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'

# 2) 上节点查 kubelet
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager
journalctl -u kubelet -f | grep -iE 'error|failed'

# 3) 容器运行时
systemctl status containerd
crictl version
crictl ps | head

# 4) 资源
df -h
free -h
cat /proc/sys/fs/file-nr

# 5) 网络与证书
ping <apiserver-ip>
curl -k https://<apiserver>:6443/healthz
ls -l /var/lib/kubelet/pki/
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

# 6) 常见错误
# "failed to run Kubelet: misconfiguration: kubelet cgroup driver: systemd is different from docker cgroup driver: cgroupfs"
#   → cgroup 驱动不一致（见 2.x / containerd 分册）
# "Unable to register node with API server" → 网络或 apiserver 不可达
# "PLEG is not healthy" → 运行时卡住，重启 containerd
# "node has insufficient memory" → 资源压力

14.3 Pod 异常排查表

# 标准五连
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --tail=200
kubectl logs <pod> -n <ns> --previous
kubectl get pod <pod> -n <ns> -o yaml
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -20

14.4 OOM 与资源类

# 确认 OOM
kubectl describe pod <pod> | grep -A3 'Last State'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# 节点侧
dmesg -T | grep -i 'killed process' | tail
journalctl -k | grep -i oom

# JVM 容器内存对齐
# 用 -XX:MaxRAMPercentage=75，不要写死 -Xmx
# 堆 + 元空间 + 直接内存 + 线程栈 ≈ limit × 0.8

# 资源使用
kubectl top pod -n <ns> --sort-by=memory
kubectl describe node <node> | grep -A10 'Allocated resources'

14.5 Service / 网络

# 1) endpoints 是否为空
kubectl get ep <svc> -n <ns>
kubectl get endpointslices -n <ns>

# 2) selector 是否匹配 pod label
kubectl get svc <svc> -n <ns> -o jsonpath='{.spec.selector}'
kubectl get pod -n <ns> --show-labels

# 3) pod 是否 Ready（未 Ready 不会进 endpoints）
kubectl get pod -n <ns> -o wide

# 4) 从 pod 内测
kubectl run -it --rm tmp --image=curlimages/curl --restart=Never -- \
  curl -v http://<svc>.<ns>.svc.cluster.local

# 5) DNS
kubectl run -it --rm dns --image=busybox:1.36 --restart=Never -- \
  nslookup <svc>.<ns>.svc.cluster.local

# 6) kube-proxy
kubectl get pod -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50

14.6 存储

kubectl get pvc -A
kubectl describe pvc <pvc> -n <ns>
kubectl get pv
kubectl get sc
kubectl get pod -A | grep csi

# 挂载失败常见事件
# "Unable to attach or mount volumes: ... timeout expired waiting for volumes to attach"
#   → CSI 驱动异常 / 节点与卷不在同 AZ
# "failed to provision volume with StorageClass" → provisioner 权限或配额

14.7 控制面与 etcd

kubectl get --raw '/readyz?verbose'
kubectl get componentstatuses        # 已弃用但仍可用
kubectl -n kube-system get pod

# etcd
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health --cluster

14.8 常用调试工具

# 临时 pod（自带网络工具）
kubectl run netshoot --rm -it --image=nicolaka/netshoot --restart=Never -- bash
# 里面：dig / curl / tcpdump / ss / nslookup / iperf3

# 节点调试
kubectl debug node/<node> -it --image=ubuntu:24.04
# 会挂载宿主根文件系统到 /host

# 复制 pod 调试（不影响原 pod）
kubectl debug <pod> --copy-to=<pod>-debug --image=ubuntu:24.04 -it

# 临时容器（1.23+，加进运行中 pod）
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>

# 资源关系树
kubectl tree deploy web -n prod

# 事件时间线
kubectl get events -n prod --sort-by=.lastTimestamp -o custom-columns=\
'TIME:.lastTimestamp,TYPE:.type,REASON:.reason,OBJ:.involvedObject.name,MSG:.message'

15. 监控与日志

15.1 metrics-server

# 安装
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 自签证书环境需加 --kubelet-insecure-tls
kubectl -n kube-system patch deploy metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

kubectl -n kube-system get pod -l k8s-app=metrics-server
kubectl top nodes
kubectl top pods -A

15.2 kube-state-metrics 与 node-exporter

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword='<pwd>' \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

关键指标（告警必配）：

15.3 日志架构

容器 stdout/stderr
   ↓
/var/log/pods/<ns>_<pod>_<uid>/<container>/0.log
   ↓ （/var/log/containers/*.log 是软链）
节点日志采集器（Fluent Bit / Filebeat / Promtail DaemonSet）
   ↓
Loki / Elasticsearch / 云日志

# 节点上看日志
ls /var/log/pods/ | head
ls -l /var/log/containers/ | head
kubectl logs <pod> --tail=100

# 日志被轮转后仍要查历史
# kubelet 默认保留 5 个 10Mi 文件
# /var/lib/kubelet/config.yaml: containerLogMaxSize / containerLogMaxFiles

# Fluent Bit DaemonSet 关键配置（简）
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector: { matchLabels: { app: fluent-bit } }
  template:
    metadata: { labels: { app: fluent-bit } }
    spec:
      serviceAccountName: fluent-bit
      tolerations: [{ operator: Exists }]
      containers:
        - name: fluent-bit
          image: cr.fluentbit.io/fluent/fluent-bit:3.1
          volumeMounts:
            - { name: varlog, mountPath: /var/log }
            - { name: containers, mountPath: /var/lib/docker/containers, readOnly: true }
            - { name: config, mountPath: /fluent-bit/etc }
      volumes:
        - { name: varlog, hostPath: { path: /var/log } }
        - { name: containers, hostPath: { path: /var/lib/docker/containers } }
        - { name: config, configMap: { name: fluent-bit-config } }

15.4 巡检脚本

#!/usr/bin/env bash
set -uo pipefail
echo "===== k8s check $(date '+%F %T') ====="

echo "--- 集群就绪"
kubectl get --raw='/readyz' && echo ok

echo "--- 节点"
kubectl get nodes -o wide

echo "--- 非 Running pod"
kubectl get pod -A --field-selector status.phase!=Running --no-headers | head -20

echo "--- 高重启 pod"
kubectl get pod -A --sort-by='.status.containerStatuses[0].restartCount' -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' --no-headers | tail -10

echo "--- 未就绪 deployment"
kubectl get deploy -A -o json | jq -r '.items[] | select(.status.unavailableReplicas>0) | .metadata.namespace+"/"+.metadata.name'

echo "--- Pending PVC"
kubectl get pvc -A --field-selector status.phase!=Bound --no-headers | head

echo "--- 近期 Warning 事件"
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp --no-headers | tail -15

echo "--- 证书"
kubeadm certs check-expiration 2>/dev/null | tail -20

echo "--- etcd"
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint health 2>/dev/null

16. 备份与迁移

16.1 备份什么

# 全量资源导出（不含 secret 内容时注意脱敏）
mkdir -p /backup/k8s/$(date +%F)
for r in ns deploy sts ds svc ing cm secret pvc pv sa role rolebinding clusterrole clusterrolebinding; do
  kubectl get $r -A -o yaml > /backup/k8s/$(date +%F)/$r.yaml 2>/dev/null
done

secret 导出是 base64 明文，注意存储加密与访问控制。

16.2 Velero（推荐的应用级备份）

# 安装 CLI
curl -fL -o velero.tar.gz https://github.com/vmware-tanzu/velero/releases/latest/download/velero-linux-amd64.tar.gz
tar xzf velero.tar.gz && mv velero-*/velero /usr/local/bin/

# 安装服务端（以 MinIO 为例）
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=false \
  --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.storage:9000

# 备份
velero backup create full-$(date +%F) --include-namespaces prod
velero backup create app-only --include-namespaces prod --exclude-resources pods
velero backup get
velero backup describe full-2026-11-05 --details
velero backup logs full-2026-11-05

# 恢复
velero restore create --from-backup full-2026-11-05
velero restore get
velero restore describe <restore-name>

# 定时备份
velero schedule create daily --schedule="0 3 * * *" --include-namespaces prod --ttl 168h
velero schedule get

# 迁移到新集群
velero backup create migrate --include-cluster-resources=true
# 新集群装同版本 velero 指向同一存储，然后 restore

16.3 集群迁移

# 方案 A：Velero（推荐，保留 PVC 依赖快照或 Restic/Kopia）
# 方案 B：纯清单迁移 + 手工搬数据
for r in ns deploy sts svc ing cm sa; do
  kubectl get $r -n prod -o yaml | kubectl --context=new apply -f -
done
# 数据：应用层 dump/restore

# 方案 C：蓝绿集群（改 DNS/LB 切换）

16.4 k3s 备份

# k3s 自动备份 SQLite 到 /var/lib/rancher/k3s/server/db/snapshots
ls /var/lib/rancher/k3s/server/db/snapshots/
k3s etcd-snapshot save --name manual-$(date +%F)
k3s etcd-snapshot ls

# 恢复
systemctl stop k3s
k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<file>
systemctl start k3s

17. 卸载与重建

卸载前：① 数据已备份（etcd 快照 + PV 数据）；② 业务已迁移；③ 停机窗口已批；④ 记录所有自定义资源清单。

17.1 完整卸载流程

# 0) 备份
kubectl get all -A -o yaml > /backup/k8s-all-$(date +%F).yaml
kubectl get pv,pvc,cm,secret -A -o yaml > /backup/k8s-storage-config.yaml
# etcd 快照见 13.2

# 1) master：驱逐并删除所有 worker
for n in $(kubectl get nodes --no-headers | awk '$3!="control-plane,master"{print $1}'); do
  kubectl drain "$n" --ignore-daemonsets --delete-emptydir-data --force
  kubectl delete node "$n"
done

# 2) 每个节点（含 master）：reset
kubeadm reset -f

每个节点必须手工清理（kubeadm reset 不做这些）

# CNI 配置与状态
rm -rf /etc/cni/net.d
rm -rf /var/lib/cni/
rm -rf /opt/cni/bin          # 若不再需要 CNI 插件

# kubelet 数据
rm -rf /var/lib/kubelet/*
rm -rf /etc/kubernetes/
rm -rf /var/lib/etcd
rm -rf /root/.kube /home/*/.kube
rm -rf /var/log/pods /var/log/containers /var/log/kubernetes

# 网卡（按实际 CNI 删）
ip link delete cni0 2>/dev/null
ip link delete flannel.1 2>/dev/null
ip link delete tunl0 2>/dev/null
ip link delete cali0 2>/dev/null
ip link show | grep -E 'cali|cilium|vxlan|flannel|cni'

# iptables / nftables
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
iptables -P INPUT ACCEPT && iptables -P FORWARD ACCEPT && iptables -P OUTPUT ACCEPT
ipvsadm --clear 2>/dev/null
nft flush ruleset 2>/dev/null     # ★ 会清掉所有防火墙规则，慎用

# 路由
ip route | grep -E '10.244|10.96|flannel|cali'

# 3) 卸载软件包
dnf remove -y kubelet kubeadm kubectl cri-tools kubernetes-cni --disableexcludes=kubernetes
rm -f /etc/yum.repos.d/kubernetes.repo
systemctl daemon-reload

# Debian
# apt-mark unhold kubelet kubeadm kubectl
# apt purge -y kubelet kubeadm kubectl

# 4) 还原系统参数
rm -f /etc/sysctl.d/99-k8s.conf
rm -f /etc/modules-load.d/k8s.conf
sysctl --system

# 5) 容器运行时是否保留？
# 保留：只清 k8s 的镜像
crictl rmi --prune
# 卸载：见 containerd 分册第 15 章

# 6) 验证
kubectl version 2>&1 | head -2
systemctl status kubelet 2>&1 | head -3
ls /etc/kubernetes 2>/dev/null
ip link | grep -E 'cni|flannel|cali'

17.2 同 IP 重建集群的坑

kubeadm reset 后不清 CNI 与网络接口，用同一 IP 重建会出现：

• pod 网段冲突（cni0 里残留旧 IP）
• Service 不通（iptables 规则混乱）
• 节点加入失败（旧证书残留）

必须执行 17.1 的清理后再 init。

17.3 只删某个命名空间

# 正常删除
kubectl delete ns prod

# 卡在 Terminating（有 finalizer 未处理）
kubectl get ns prod -o json | jq '.spec.finalizers'
kubectl get ns prod -o json | jq '.spec.finalizers=[]' | kubectl replace --raw "/api/v1/namespaces/prod/finalize" -f -

# 仍有资源残留时逐个清
kubectl api-resources --verbs=list --namespaced -o name | \
  xargs -n1 kubectl get -n prod --show-kind --ignore-not-found

18. 高频故障速查

18.1 集群级

18.2 调度与资源

18.3 网络

18.4 存储

18.5 应用层

19. 附录

19.1 常用资源缩写

19.2 端口

19.3 关键路径

19.4 最小可用清单

# 一个能跑的最小 Deployment + Service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 2
  selector: { matchLabels: { app: demo } }
  template:
    metadata: { labels: { app: demo } }
    spec:
      containers:
        - name: demo
          image: nginx:1.25
          ports: [{ containerPort: 80 }]
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { cpu: 500m, memory: 256Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  selector: { app: demo }
  ports: [{ port: 80, targetPort: 80 }]

19.5 与其他手册的配合

• 容器运行时（含 CRI、镜像拉取、SystemdCgroup）：containerd-常用运维指令手册.md
• 单机容器与镜像构建：Docker-常用运维指令手册.md / Podman-常用运维指令手册.md
• 私有镜像仓库：Harbor-镜像仓库运维指令手册.md
• 宿主系统（内核、cgroup、SELinux、磁盘）：CentOS-7-10-常规运维指令手册.md / Ubuntu-常用运维指令手册.md
• etcd 底层与数据库运维思路：MySQL-常用运维指令手册.md（备份/恢复方法论可借鉴）

一句话总结：Kubernetes 的运维难点不在命令，而在排查路径——先分层（集群/节点/工作负载/容器/应用），再看事件（describe 的 Events 覆盖 80% 问题），最后才进容器。把 kubectl get --raw /readyz?verbose、kubectl describe、kubectl logs --previous 这三招用熟，大部分故障能在 5 分钟内定位。