规划
三台机器
172.16.10.11 部署单机版k8s，部署主jenkins与一个agent
172.16.10.12 裸部署gitlab
172.16.10.13 充当开发人员的开发机，用于上传代码
部署k8s集群
官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
kubeadm部署k8s高可用集群的官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
（1）基本配置
注意：
确保CPU至少2核
内存的话至少1.5G，推荐2-3G
（2）在所有主机执行如下操作
1、配置静态ip地址
略
2、 每台主机都关闭NetworkManager
systemctl stop NetworkManager systemctl disable NetworkManager
3、每台主机均关闭selinux与防火墙
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config setenforce 0 systemctl stop firewalld.service systemctl disable firewalld.service
4、关闭swap分区
# Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动,所以我们有两种处理方式，采用一种即可 方式一：关闭swap分区 swapoff -a # 先临时关闭，立即生效 sed -i 's/.*swap.*/#&/' /etc/fstab # 注释掉swap，永久关闭，保证即便重启主机也会生效 方式二： kubelet忽略swap echo 'KUBELET_EXTRA_ARGS="--fail-swap-on=false"' > /etc/sysconfig/kubelet
5、为三台主机配置各自的主机名
hostnamectl set-hostname master hostnamectl set-hostname node01 hostnamectl set-hostname node02
6、添加解析
cat >> /etc/hosts << EOF 172.16.10.14 master 172.16.10.15 node01 172.16.10.16 node02 EOF
7、每台机器均修改ssh配置
加快远程链接速度，可选，但建议做 sed -ri '/#UseDNS yes/c UseDNS no' /etc/ssh/sshd_config systemctl restart sshd
8、做免密登录（只有主节点做，此步为建议步骤，非必须）
[root@master ~]# ssh-keygen [root@master ~]# ssh-copy-id -i root@master [root@master ~]# ssh-copy-id -i root@node01 [root@master ~]# ssh-copy-id -i root@node02
9、更新系统软件(排除内核)
yum install epel-release -y && yum update -y --exclud=kernel*
10、安装基础常用软件
yum install wget expect vim net-tools ntp bash-completion ipvsadm ipset jq iptables conntrack sysstat libseccomp -y # 其他（选做） yum -y install python-setuptools python-pip gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel \ zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel zip unzip ncurses ncurses-devel curl curl-devel e2fsprogs \ e2fsprogs-devel krb5-devel libidn libidn-devel openssl openssh openssl-devel nss_ldap openldap openldap-devel openldap-clients \ openldap-servers libxslt-devel libevent-devel ntp libtool-ltdl bison libtool vim-enhanced python wget lsof iptraf strace lrzsz \ kernel-devel kernel-headers pam-devel tcl tk cmake ncurses-devel bison setuptool popt-devel net-snmp screen perl-devel \ pcre-devel net-snmp screen tcpdump rsync sysstat man iptables sudo libconfig git bind-utils \ tmux elinks numactl iftop bwm-ng net-tools expect
11、更新系统内核（docker 对系统内核要求比较高，最好使用4.4+），非必须操作，推荐做
一般来说，只有从https://www.kernel.org/ 下载并编译安装的内核才是官方内核,可以看出目前的稳定版版本为5.18.10
不过，大多数 Linux 发行版提供自行维护的内核，可以通过 yum 或 rpm 等包管理系统升级。
ELRepo是一个为Linux提供驱动程序和内核映像的存储库，这里的升级方案就是采用ELRepo提供的内核通道。
ELRepo官网：http://elrepo.org/tiki/tiki-index.php
# 1、升级系统内核 #查看 yum 中可升级的内核版本 yum list kernel --showduplicates #如果list中有需要的版本可以直接执行 update 升级，多数是没有的，所以要按以下步骤操作 #导入ELRepo软件仓库的公共秘钥 rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org #Centos7系统安装ELRepo yum -y install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm #Centos8系统安装ELRepo #yum -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm #查看ELRepo提供的内核版本 yum --disablerepo="*" --enablerepo="elrepo-kernel" list available #kernel-lt：表示longterm，即长期支持的内核；当前为5.4. #kernel-ml：表示mainline，即当前主线的内核；当前为5.17. #安装主线内核 yum --enablerepo=elrepo-kernel install kernel-ml.x86_64 -y #查看系统可用内核，并设置启动项 sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg #0 : CentOS Linux (5.17.1-1.el7.elrepo.x86_64) 7 (Core) #1 : CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core) #2 : CentOS Linux (3.10.0-1160.el7.x86_64) 7 (Core) #3 : CentOS Linux (0-rescue-20220208145000711038896885545492) 7 (Core) #指定开机启动内核版本 grub2-set-default 0 # 或者 grub2-set-default 'CentOS Linux (5.17.1-1.el7.elrepo.x86_64) 7 (Core)' #生成 grub 配置文件 grub2-mkconfig -o /boot/grub2/grub.cfg #查看当前默认启动的内核 grubby --default-kernel #重启系统，验证 uname -r
8、配置ntp服务，保证集群服务器时间统一(统一时间非常重要，必须要做)
# 大前提：chrony服务端客户端配置完后，重启chronyd服务即可快速完成时间同步，在这之后就不要再手动去修改时间了，一切让时间服务器自己去同步 # ====================>chrony服务端：master节点<==================== # 1、安装 yum -y install chrony # 2、修改配置文件 mv /etc/chrony.conf /etc/chrony.conf.bak cat > /etc/chrony.conf << EOF server ntp1.aliyun.com iburst minpoll 4 maxpoll 10 server ntp2.aliyun.com iburst minpoll 4 maxpoll 10 server ntp3.aliyun.com iburst minpoll 4 maxpoll 10 server ntp4.aliyun.com iburst minpoll 4 maxpoll 10 server ntp5.aliyun.com iburst minpoll 4 maxpoll 10 server ntp6.aliyun.com iburst minpoll 4 maxpoll 10 server ntp7.aliyun.com iburst minpoll 4 maxpoll 10 driftfile /var/lib/chrony/drift makestep 10 3 rtcsync allow 0.0.0.0/0 local stratum 10 keyfile /etc/chrony.keys logdir /var/log/chrony stratumweight 0.05 noclientlog logchange 0.5 EOF # 4、启动chronyd服务 systemctl restart chronyd.service # 最好重启，这样无论原来是否启动都可以重新加载配置 systemctl enable chronyd.service systemctl status chronyd.service ====================>chrony客户端：其他节点，完全一样的配置与操作<==================== # 下述步骤一次性粘贴到每个客户端执行即可 # 1、安装chrony yum -y install chrony # 2、需改客户端配置文件 mv /etc/chrony.conf /etc/chrony.conf.bak cat > /etc/chrony.conf << EOF server master iburst driftfile /var/lib/chrony/drift makestep 10 3 rtcsync local stratum 10 keyfile /etc/chrony.key logdir /var/log/chrony stratumweight 0.05 noclientlog logchange 0.5 EOF # 3、启动chronyd systemctl restart chronyd.service systemctl enable chronyd.service systemctl status chronyd.service # 4、验证 chronyc sources -v
7、安装docker
# 1、选做，卸载之前的docker yum -y remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-selinux \ docker-engine-selinux \ docker-engine # 2、安装docker所需安装包 yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install docker-ce -y # 3、启动并设置开机启动 systemctl start docker && systemctl enable docker && systemctl status docker # 4、基本配置 cat > /etc/docker/daemon.json << EOF { "exec-opts": ["native.cgroupdriver=systemd"], "registry-mirrors":["https://reg-mirror.qiniu.com/"], "live-restore":true } EOF # 5、重启 systemctl restart docker && docker info
8、拉取镜像（再次强调，每台机器都需要操作）
kubeadm部署时会去指定的地址拉取镜像，该地址在墙外无法访问，所以我们从阿里云拉取，并tag为指定的地址即可
#1、=====>编写脚本 cat > dockpullImages1.18.1.sh << EOF #!/bin/bash ##所需要的镜像名字 #k8s.gcr.io/kube-apiserver:v1.18.1 #k8s.gcr.io/kube-controller-manager:v1.18.1 #k8s.gcr.io/kube-scheduler:v1.18.1 #k8s.gcr.io/kube-proxy:v1.18.1 #k8s.gcr.io/pause:3.2 #k8s.gcr.io/etcd:3.4.3-0 #k8s.gcr.io/coredns:1.6.7 ###拉取镜像 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.1 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.1 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.1 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.1 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 ###修改tag docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.1 k8s.gcr.io/kube-apiserver:v1.18.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.1 k8s.gcr.io/kube-controller-manager:v1.18.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.1 k8s.gcr.io/kube-scheduler:v1.18.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.1 k8s.gcr.io/kube-proxy:v1.18.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 k8s.gcr.io/coredns:1.6.7 EOF # 2、在所有节点执行该脚本 sh dockpullImages1.18.1.sh
9、安装kubelet、kubeadm 和 kubectl（所有节点执行）
kubelet 运行在 Cluster 所有节点上，负责启动 Pod 和容器。
kubeadm 用于初始化 Cluster。
kubectl 是 Kubernetes 命令行工具。通过 kubectl 可以部署和管理应用，查看各种资源，创建、删除和更新各种组件。
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF sed -ri 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/kubernetes.repo
在所有节点安装
1.安装 yum makecache fast # yum install -y kubelet kubeadm kubectl ipvsadm #注意，这样默认是下载最新版本v1.22.2 ====================================================================== [root@master ~]# yum install -y kubelet-1.18.1-0.x86_64 kubeadm-1.18.1-0.x86_64 kubectl-1.18.1-0.x86_64 ipvsadm 2.加载ipvs相关内核模块 yum install -y conntrack-tools ipvsadm ipvsadmin ipset conntrack libseccomp 如果重新开机，需要重新加载（可以写在 /etc/rc.local 中开机自动加载） modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh #modprobe nf_conntrack_ipv4 # 如果是3.x内核，那么应该加载这一样 modprobe nf_conntrack # 如果是高版本内核比如5.x，那么应该加载这个。在高版本内核已经把nf_conntrack_ipv4替换为nf_conntrack了。 3.编辑文件添加开机启动 cat >> /etc/rc.local << EOF modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack #modprobe nf_conntrack_ipv4 EOF chmod +x /etc/rc.local 重启服务器 reboot 4.配置： 配置转发相关参数，否则可能会出错 cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness=0 EOF 5.使配置生效 sysctl --system 6.如果net.bridge.bridge-nf-call-iptables报错，加载br_netfilter模块 # modprobe br_netfilter # sysctl -p /etc/sysctl.d/k8s.conf 7.查看是否加载成功 [root@master ~]# lsmod | grep ip_vs ip_vs_sh 16384 0 ip_vs_wrr 16384 0 ip_vs_rr 16384 0 ip_vs 159744 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 151552 5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs
10、启动kubelet
#1.配置kubelet使用pause镜像 #配置变量： systemctl start docker && systemctl enable docker DOCKER_CGROUPS=$(docker info | grep 'Cgroup Driver' | cut -d' ' -f4) echo $DOCKER_CGROUPS #这个是使用国内的源。-###注意我们使用谷歌的镜像--操作下面的第3标题 #2.配置kubelet的cgroups cat >/etc/sysconfig/kubelet<<EOF KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2" EOF #cat >/etc/sysconfig/kubelet<<EOF #KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs --pod-infra-container-image=k8s.gcr.io/pause:3.2" #EOF
启动
systemctl daemon-reload systemctl enable kubelet && systemctl restart kubelet # 注意在这里使用 # systemctl status kubelet，你会发现报错误信息； # 7月 10 23:28:36 master systemd[1]: Unit kubelet.service entered failed state. # 7月 10 23:28:36 master systemd[1]: kubelet.service failed. #运行 # journalctl -xefu kubelet 命令查看systemd日志会发现提示缺少一些问题件 #这个错误在运行kubeadm init 生成CA证书后会被自动解决，此处可先忽略。 #简单地说就是在kubeadm init 之前kubelet会不断重启。
11、初始化master
kubeadm init \ --kubernetes-version=v1.18.1 \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=172.16.10.14 \ --ignore-preflight-errors=Swap
注意修改apiserver-advertise-address为master节点ip
参数解释：
–kubernetes-version: 用于指定k8s版本； –apiserver-advertise-address：用于指定kube-apiserver监听的ip地址,就是 master本机IP地址。 –pod-network-cidr：用于指定Pod的网络范围； 10.244.0.0/16 –service-cidr：用于指定SVC的网络范围； –image-repository: 指定阿里云镜像仓库地址
看到以下信息表示安装成功
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \ --discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7
成功后注意最后一个命令，这个join命令可以用来添加节点。
注意保持好kubeadm join，后面会用到的。
如果初始化失败，请使用如下代码清除后重新初始化
# kubeadm reset
12、按照提示配置kubectl
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # 查看 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
13、配置使用网络插件
要让 Kubernetes Cluster 能够工作，必须安装 Pod 网络，否则 Pod 之间无法通信。
Kubernetes 支持多种网络方案，这里我们先使用 flannel，后面还会讨论 Canal。
flannel.yaml
--- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny' --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: registry.cn-hangzhou.aliyuncs.com/alvinos/flanned:v0.13.1-rc1 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: registry.cn-hangzhou.aliyuncs.com/alvinos/flanned:v0.13.1-rc1 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg
部署flannel
kubectl apply -f flannel.yaml
查看
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 6m49s v1.18.1 [root@master ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-66bff467f8-mtrkv 1/1 Running 0 3m57s coredns-66bff467f8-qhwv4 1/1 Running 0 3m57s etcd-master 1/1 Running 0 4m9s kube-apiserver-master 1/1 Running 0 4m9s kube-controller-manager-master 1/1 Running 0 4m9s kube-flannel-ds-b25g6 1/1 Running 0 32s kube-proxy-gkqjv 1/1 Running 0 3m57s kube-scheduler-master 1/1 Running 0 4m9s
14、添加node01与node02
在所有node节点执行，登录到node节点，确保已经安装了docker和kubeadm，kubelet，kubectl
如果报错开启ip转发： # sysctl -w net.ipv4.ip_forward=1 在所有node节点操作，此命令为初始化master成功后返回的结果 kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \ --discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7 # 在master节点查看，最开始两个加入的节点是NoReady，过一会后处于Ready状态 [root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 7m45s v1.18.1 node01 Ready <none> 25s v1.18.1 node02 Ready <none> 22s v1.18.1
15、kubeadm init创建完集群后，有pod一直是pending状态
kubectl describe pod如果发现问题 3 node(s) had taints that the pod didn't tolerate. kubernetes出于安全考虑默认情况下无法在master节点上部署pod，于是用下面方法去掉master节点的污点： kubectl taint nodes --all node-role.kubernetes.io/master-
16、了解：
# 1、移除node方法 kubectl drain node02 --delete-local-data --force --ignore-daemonsets kubectl delete nodes node02 # 2、添加已删除节点 前提：token未失效 如果这个时候再想添加进来这个node，需要执行两步操作 第一步：停掉kubelet(需要添加进来的节点操作) [root@node02 ~]# systemctl stop kubelet 第二步：删除相关文件 [root@node02 ~]# rm -rf /etc/kubernetes/* 第三步：添加节点 因为之前的token还有效，我这里并没有超出token的有效期；直接执行加入集群的命令即可； kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \ --discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7 # 3、忘记token了，怎么办，在master节点 [root@master ~]# kubeadm token list # 可以拿到token值，然后再用openssl解析出hash值就行 TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS n3mvgw.56ul27rjtox7fr3n 23h 2022-07-11T23:32:18+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token [root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7 然后重新拼接命令kubeadm join就行 kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n --discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7
