规划
三台机器
172.16.10.11 部署单机版k8s，部署主jenkins与一个agent
172.16.10.12 裸部署gitlab
172.16.10.13 充当开发人员的开发机，用于上传代码
部署k8s集群
官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
kubeadm部署k8s高可用集群的官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
（1）基本配置
注意：
确保CPU至少2核
内存的话至少1.5G，推荐2-3G
（2）在所有主机执行如下操作
1、配置静态ip地址
略
2、 每台主机都关闭NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager3、每台主机均关闭selinux与防火墙
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0
systemctl stop firewalld.service
systemctl disable firewalld.service4、关闭swap分区
# Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动,所以我们有两种处理方式，采用一种即可
方式一：关闭swap分区
swapoff -a # 先临时关闭，立即生效
sed -i 's/.*swap.*/#&/' /etc/fstab # 注释掉swap，永久关闭，保证即便重启主机也会生效
方式二： kubelet忽略swap
echo 'KUBELET_EXTRA_ARGS="--fail-swap-on=false"' > /etc/sysconfig/kubelet5、为三台主机配置各自的主机名
hostnamectl set-hostname master
hostnamectl set-hostname node01
hostnamectl set-hostname node026、添加解析
cat >> /etc/hosts << EOF
172.16.10.14 master
172.16.10.15 node01
172.16.10.16 node02
EOF7、每台机器均修改ssh配置
加快远程链接速度，可选，但建议做
sed -ri '/#UseDNS yes/c UseDNS no' /etc/ssh/sshd_config
systemctl restart sshd8、做免密登录（只有主节点做，此步为建议步骤，非必须）
[root@master ~]# ssh-keygen
[root@master ~]# ssh-copy-id -i root@master
[root@master ~]# ssh-copy-id -i root@node01
[root@master ~]# ssh-copy-id -i root@node029、更新系统软件(排除内核)
yum install epel-release -y && yum update -y --exclud=kernel*10、安装基础常用软件
yum install wget expect vim net-tools ntp bash-completion ipvsadm ipset jq iptables conntrack sysstat libseccomp -y
# 其他（选做）
yum -y install python-setuptools python-pip gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel \
zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel zip unzip ncurses ncurses-devel curl curl-devel e2fsprogs \
e2fsprogs-devel krb5-devel libidn libidn-devel openssl openssh openssl-devel nss_ldap openldap openldap-devel openldap-clients \
openldap-servers libxslt-devel libevent-devel ntp libtool-ltdl bison libtool vim-enhanced python wget lsof iptraf strace lrzsz \
kernel-devel kernel-headers pam-devel tcl tk cmake ncurses-devel bison setuptool popt-devel net-snmp screen perl-devel \
pcre-devel net-snmp screen tcpdump rsync sysstat man iptables sudo libconfig git bind-utils \
tmux elinks numactl iftop bwm-ng net-tools expect11、更新系统内核（docker 对系统内核要求比较高，最好使用4.4+），非必须操作，推荐做
一般来说，只有从https://www.kernel.org/ 下载并编译安装的内核才是官方内核,可以看出目前的稳定版版本为5.18.10
不过，大多数 Linux 发行版提供自行维护的内核，可以通过 yum 或 rpm 等包管理系统升级。
ELRepo是一个为Linux提供驱动程序和内核映像的存储库，这里的升级方案就是采用ELRepo提供的内核通道。
ELRepo官网：http://elrepo.org/tiki/tiki-index.php
# 1、升级系统内核
#查看 yum 中可升级的内核版本
yum list kernel --showduplicates
#如果list中有需要的版本可以直接执行 update 升级，多数是没有的，所以要按以下步骤操作
#导入ELRepo软件仓库的公共秘钥
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
#Centos7系统安装ELRepo
yum -y install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
#Centos8系统安装ELRepo
#yum -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
#查看ELRepo提供的内核版本
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
#kernel-lt：表示longterm，即长期支持的内核；当前为5.4.
#kernel-ml：表示mainline，即当前主线的内核；当前为5.17.
#安装主线内核
yum --enablerepo=elrepo-kernel install kernel-ml.x86_64 -y
#查看系统可用内核，并设置启动项
sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
#0 : CentOS Linux (5.17.1-1.el7.elrepo.x86_64) 7 (Core)
#1 : CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core)
#2 : CentOS Linux (3.10.0-1160.el7.x86_64) 7 (Core)
#3 : CentOS Linux (0-rescue-20220208145000711038896885545492) 7 (Core)
#指定开机启动内核版本
grub2-set-default 0 # 或者 grub2-set-default 'CentOS Linux (5.17.1-1.el7.elrepo.x86_64) 7 (Core)'
#生成 grub 配置文件
grub2-mkconfig -o /boot/grub2/grub.cfg
#查看当前默认启动的内核
grubby --default-kernel
#重启系统，验证
uname -r8、配置ntp服务，保证集群服务器时间统一(统一时间非常重要，必须要做)
# 大前提：chrony服务端客户端配置完后，重启chronyd服务即可快速完成时间同步，在这之后就不要再手动去修改时间了，一切让时间服务器自己去同步
# ====================>chrony服务端：master节点<====================
# 1、安装
yum -y install chrony
# 2、修改配置文件
mv /etc/chrony.conf /etc/chrony.conf.bak
cat > /etc/chrony.conf << EOF
server ntp1.aliyun.com iburst minpoll 4 maxpoll 10
server ntp2.aliyun.com iburst minpoll 4 maxpoll 10
server ntp3.aliyun.com iburst minpoll 4 maxpoll 10
server ntp4.aliyun.com iburst minpoll 4 maxpoll 10
server ntp5.aliyun.com iburst minpoll 4 maxpoll 10
server ntp6.aliyun.com iburst minpoll 4 maxpoll 10
server ntp7.aliyun.com iburst minpoll 4 maxpoll 10
driftfile /var/lib/chrony/drift
makestep 10 3
rtcsync
allow 0.0.0.0/0
local stratum 10
keyfile /etc/chrony.keys
logdir /var/log/chrony
stratumweight 0.05
noclientlog
logchange 0.5
EOF
# 4、启动chronyd服务
systemctl restart chronyd.service # 最好重启，这样无论原来是否启动都可以重新加载配置
systemctl enable chronyd.service
systemctl status chronyd.service
====================>chrony客户端：其他节点，完全一样的配置与操作<====================
# 下述步骤一次性粘贴到每个客户端执行即可
# 1、安装chrony
yum -y install chrony
# 2、需改客户端配置文件
mv /etc/chrony.conf /etc/chrony.conf.bak
cat > /etc/chrony.conf << EOF
server master iburst
driftfile /var/lib/chrony/drift
makestep 10 3
rtcsync
local stratum 10
keyfile /etc/chrony.key
logdir /var/log/chrony
stratumweight 0.05
noclientlog
logchange 0.5
EOF
# 3、启动chronyd
systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service
# 4、验证
chronyc sources -v7、安装docker
# 1、选做，卸载之前的docker
yum -y remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
# 2、安装docker所需安装包
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce -y
# 3、启动并设置开机启动
systemctl start docker && systemctl enable docker && systemctl status docker
# 4、基本配置
cat > /etc/docker/daemon.json << EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors":["https://reg-mirror.qiniu.com/"],
"live-restore":true
}
EOF
# 5、重启
systemctl restart docker && docker info8、拉取镜像（再次强调，每台机器都需要操作）
kubeadm部署时会去指定的地址拉取镜像，该地址在墙外无法访问，所以我们从阿里云拉取，并tag为指定的地址即可
#1、=====>编写脚本
cat > dockpullImages1.18.1.sh << EOF
#!/bin/bash
##所需要的镜像名字
#k8s.gcr.io/kube-apiserver:v1.18.1
#k8s.gcr.io/kube-controller-manager:v1.18.1
#k8s.gcr.io/kube-scheduler:v1.18.1
#k8s.gcr.io/kube-proxy:v1.18.1
#k8s.gcr.io/pause:3.2
#k8s.gcr.io/etcd:3.4.3-0
#k8s.gcr.io/coredns:1.6.7
###拉取镜像
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7
###修改tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.1 k8s.gcr.io/kube-apiserver:v1.18.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.1 k8s.gcr.io/kube-controller-manager:v1.18.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.1 k8s.gcr.io/kube-scheduler:v1.18.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.1 k8s.gcr.io/kube-proxy:v1.18.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 k8s.gcr.io/coredns:1.6.7
EOF
# 2、在所有节点执行该脚本
sh dockpullImages1.18.1.sh9、安装kubelet、kubeadm 和 kubectl（所有节点执行）
kubelet 运行在 Cluster 所有节点上，负责启动 Pod 和容器。
kubeadm 用于初始化 Cluster。
kubectl 是 Kubernetes 命令行工具。通过 kubectl 可以部署和管理应用，查看各种资源，创建、删除和更新各种组件。
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -ri 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/kubernetes.repo 在所有节点安装
1.安装
yum makecache fast
# yum install -y kubelet kubeadm kubectl ipvsadm #注意，这样默认是下载最新版本v1.22.2
======================================================================
[root@master ~]# yum install -y kubelet-1.18.1-0.x86_64 kubeadm-1.18.1-0.x86_64 kubectl-1.18.1-0.x86_64 ipvsadm
2.加载ipvs相关内核模块
yum install -y conntrack-tools ipvsadm ipvsadmin ipset conntrack libseccomp
如果重新开机，需要重新加载（可以写在 /etc/rc.local 中开机自动加载）
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
#modprobe nf_conntrack_ipv4 # 如果是3.x内核，那么应该加载这一样
modprobe nf_conntrack # 如果是高版本内核比如5.x，那么应该加载这个。在高版本内核已经把nf_conntrack_ipv4替换为nf_conntrack了。
3.编辑文件添加开机启动
cat >> /etc/rc.local << EOF
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack
#modprobe nf_conntrack_ipv4
EOF
chmod +x /etc/rc.local
重启服务器 reboot
4.配置：
配置转发相关参数，否则可能会出错
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
5.使配置生效
sysctl --system
6.如果net.bridge.bridge-nf-call-iptables报错，加载br_netfilter模块
# modprobe br_netfilter
# sysctl -p /etc/sysctl.d/k8s.conf
7.查看是否加载成功
[root@master ~]# lsmod | grep ip_vs
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 159744 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 151552 5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs
libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs10、启动kubelet
#1.配置kubelet使用pause镜像
#配置变量：
systemctl start docker && systemctl enable docker
DOCKER_CGROUPS=$(docker info | grep 'Cgroup Driver' | cut -d' ' -f4)
echo $DOCKER_CGROUPS
#这个是使用国内的源。-###注意我们使用谷歌的镜像--操作下面的第3标题
#2.配置kubelet的cgroups
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF
#cat >/etc/sysconfig/kubelet<<EOF
#KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs --pod-infra-container-image=k8s.gcr.io/pause:3.2"
#EOF启动
systemctl daemon-reload
systemctl enable kubelet && systemctl restart kubelet
# 注意在这里使用 # systemctl status kubelet，你会发现报错误信息；
# 7月 10 23:28:36 master systemd[1]: Unit kubelet.service entered failed state.
# 7月 10 23:28:36 master systemd[1]: kubelet.service failed.
#运行 # journalctl -xefu kubelet 命令查看systemd日志会发现提示缺少一些问题件
#这个错误在运行kubeadm init 生成CA证书后会被自动解决，此处可先忽略。
#简单地说就是在kubeadm init 之前kubelet会不断重启。11、初始化master
kubeadm init \
--kubernetes-version=v1.18.1 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=172.16.10.14 \
--ignore-preflight-errors=Swap注意修改apiserver-advertise-address为master节点ip
参数解释：
–kubernetes-version: 用于指定k8s版本；
–apiserver-advertise-address：用于指定kube-apiserver监听的ip地址,就是 master本机IP地址。
–pod-network-cidr：用于指定Pod的网络范围； 10.244.0.0/16
–service-cidr：用于指定SVC的网络范围；
–image-repository: 指定阿里云镜像仓库地址看到以下信息表示安装成功
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \
--discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7 成功后注意最后一个命令，这个join命令可以用来添加节点。
注意保持好kubeadm join，后面会用到的。
如果初始化失败，请使用如下代码清除后重新初始化
# kubeadm reset12、按照提示配置kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 查看
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config13、配置使用网络插件
要让 Kubernetes Cluster 能够工作，必须安装 Pod 网络，否则 Pod 之间无法通信。
Kubernetes 支持多种网络方案，这里我们先使用 flannel，后面还会讨论 Canal。
flannel.yaml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: registry.cn-hangzhou.aliyuncs.com/alvinos/flanned:v0.13.1-rc1
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: registry.cn-hangzhou.aliyuncs.com/alvinos/flanned:v0.13.1-rc1
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg部署flannel
kubectl apply -f flannel.yaml查看
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 6m49s v1.18.1
[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-mtrkv 1/1 Running 0 3m57s
coredns-66bff467f8-qhwv4 1/1 Running 0 3m57s
etcd-master 1/1 Running 0 4m9s
kube-apiserver-master 1/1 Running 0 4m9s
kube-controller-manager-master 1/1 Running 0 4m9s
kube-flannel-ds-b25g6 1/1 Running 0 32s
kube-proxy-gkqjv 1/1 Running 0 3m57s
kube-scheduler-master 1/1 Running 0 4m9s14、添加node01与node02
在所有node节点执行，登录到node节点，确保已经安装了docker和kubeadm，kubelet，kubectl
如果报错开启ip转发：
# sysctl -w net.ipv4.ip_forward=1
在所有node节点操作，此命令为初始化master成功后返回的结果
kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \
--discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7
# 在master节点查看，最开始两个加入的节点是NoReady，过一会后处于Ready状态
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 7m45s v1.18.1
node01 Ready <none> 25s v1.18.1
node02 Ready <none> 22s v1.18.115、kubeadm init创建完集群后，有pod一直是pending状态
kubectl describe pod如果发现问题
3 node(s) had taints that the pod didn't tolerate.
kubernetes出于安全考虑默认情况下无法在master节点上部署pod，于是用下面方法去掉master节点的污点：
kubectl taint nodes --all node-role.kubernetes.io/master-16、了解：
# 1、移除node方法
kubectl drain node02 --delete-local-data --force --ignore-daemonsets
kubectl delete nodes node02
# 2、添加已删除节点
前提：token未失效
如果这个时候再想添加进来这个node，需要执行两步操作
第一步：停掉kubelet(需要添加进来的节点操作)
[root@node02 ~]# systemctl stop kubelet
第二步：删除相关文件
[root@node02 ~]# rm -rf /etc/kubernetes/*
第三步：添加节点
因为之前的token还有效，我这里并没有超出token的有效期；直接执行加入集群的命令即可；
kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n \
--discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7
# 3、忘记token了，怎么办，在master节点
[root@master ~]# kubeadm token list # 可以拿到token值，然后再用openssl解析出hash值就行
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
n3mvgw.56ul27rjtox7fr3n 23h 2022-07-11T23:32:18+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
[root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7
然后重新拼接命令kubeadm join就行
kubeadm join 172.16.10.14:6443 --token n3mvgw.56ul27rjtox7fr3n --discovery-token-ca-cert-hash sha256:b93e84284d278e4056ee5a1b0370a20f49a1878df8a3492f5f855e2d5141e6e7