文章目录

• 实验环境
• 实验过程
• • 1.Etcd集群部署
  • 2.docker引擎部署
  • 3.flannel网络部署
  • 4.部署master节点
  • 5.部署node节点
• 实验故障及如何处理

实验环境

主机名       IP地址       安装软件
Master01：14.0.0.50    kube-apiserver kube-controller-manager kube-scheduler etcd
Node01：  14.0.0.60    kubelet kube-proxy docker flannel etcd
Node02：  14.0.0.70    kubelet kube-proxy docker flannel etcd

实验过程

1.Etcd集群部署

#初始化环境，准备制作证书

[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls      #从宿主机上传进来
etcd-cert.sh etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert/

1.下载证书制作工具

[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

2.执行脚本下载 cfssl官方包

[root@localhost k8s]# bash cfssl.sh
[root@localhost k8s]# ls /usr/local/bin/
cfssl cfssl-certinfo cfssljson
#cfssl：生成证书工具 
#cfssljson：通过传入json文件生成证书
#cfssl-certinfo：查看证书信息

3.开始制作证书

（1）创建ca配置文件
cat > ca-config.json <<EOF
{ 
  "signing": { 
     "default": { 
        "expiry": "87600h"
    },
    "profiles": { 
    "www": { 
        "expiry": "87600h",
       "usages": [
           "signing",
           "key encipherment",
           "server auth",
           "client auth"
        ]
     }
   }
  }
}
EOF
（2）创建ca证书签名请求
cat > ca-csr.json <<EOF
{ 
     "CN": "etcd CA",
     "key": { 
           "algo": "rsa",
           "size": 2048
 },
 "names": [
       { 
       "C": "CN",
       "L": "Beijing",
       "ST": "Beijing"
         }
     ]
}
EOF

（3）生成证书
ca-key.pem：根证书的私钥
ca.pem：ca根证书文件
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
（4）指定 etcd三个节点之间的通信验证
cat > server-csr.json <<EOF
{ 
      "CN": "etcd",
      "hosts": [
      "14.0.0.50",
     "14.0.0.60",
     "14.0.0.70"
     ],
     "key": { 
     "algo": "rsa",
     "size": 2048
     },
  "names": [
         { 
           "C": "CN",
           "L": "BeiJing",
           "ST": "BeiJing"
       }
    ]
}
EOF
（5）生成 ETCD证书 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

#证书生成完毕，开始配置etcd软件

#ETCD 二进制包地址
https://github.com/etcd-io/etcd/releases
将etcd-v3.3.10-linux-amd64.tar.gz上传到/root/k8s目录下
[root@localhost k8s]# ls
cfssl.sh etcd.sh
etcd-cert etcd-v3.3.10-linux-amd64.tar.gz
（1）将etcd软件包进行解压
[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
（2）创建etcd专门的配置文件，命令文件和证书存放目录
[root@localhost k8s]# mkdir -p /opt/etcd/{ cfg,bin,ssl} 
（3）将etcd软件包中的命令文件移动到/opt/etcd/bin目录下
[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
（4）拷贝证书到/opt/etcd/ssl目录下
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/ #启动etcd.sh脚本生成配置文件和服务启动脚本；并启动etcd服务 [root@localhost k8s]# bash etcd.sh etcd01 14.0.0.50 etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380 #这时会等待其他节点加入 #使用另外一个会话打开，发现 etcd进程已经开启 [root@localhost ~]# ps -ef | grep etcd 

#使用scp拷贝配置文件，命令文件，证书文件到两个node节点
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.60:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.70:/opt
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.60:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.70:/usr/lib/systemd/system/

#进入node01节点修改（node02节点同理）
[root@localhost ~]# vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://14.0.0.60:2380"   #群集内部通信端口
ETCD_LISTEN_CLIENT_URLS="https://14.0.0.60:2379"   #对外提供服务端口
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://14.0.0.60:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://14.0.0.60:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://14.0.0.50:2380,etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#启动etcd
[root@localhost ssl]# systemctl start etcd
[root@localhost ssl]# systemctl status etcd

#使用etcdctl命令检查群集健康状态
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" cluster-health
member 3eae9a550e2e3ec is healthy: got healthy result from https://192.168.195.151:2379
member 26cd4dcf17bc5cbd is healthy: got healthy result from https://192.168.195.150:2379
member 2fcd2df8a9411750 is healthy: got healthy result from https://192.168.195.149:2379
cluster is healthy     #说明群集健康

2.docker引擎部署

所有node节点部署docker-ce引擎
部署docker-ce社区版可参考我之前的博客：
https://blog.csdn.net/chengu04/article/details/108723407

3.flannel网络部署

所有node节点部署flannel组件

（1）写入分配的子网段到etcd中，供flannel使用（注意：要在/opt/etcd/ssl目录下）
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{  "Network": "172.17.0.0/16", "Backend": { "Type": "vxlan"}}

（2）在另一个节点查看写入信息
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" get /coreos.com/network/config
{  "Network": "172.17.0.0/16", "Backend": { "Type": "vxlan"}}
（3）在两个node节点上部署flannel软件
[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md
#创建k8s工作目录，将软件包中的命令文件移动到/opt/kubernetes/bin/目录下
[root@localhost ~]# mkdir -p /opt/kubernetes/{ cfg,bin,ssl} 
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
#通过脚本配置flannel
[root@localhost ~]# vim flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${ 1:-"http://127.0.0.1:2379"}
cat <<EOF >/opt/kubernetes/cfg/flanneld        #生成配置文件
FLANNEL_OPTIONS="-- etcd-endpoints=${ETCD_ENDPOINTS} \ -etcd-cafile=/opt/etcd/ssl/ca.pem \ -etcd-certfile=/opt/etcd/ssl/server.pem \ -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/flanneld.service          #生成服务启动脚本
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld -- ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld


#开始flannel网络功能
[root@localhost ~]# bash flannel.sh https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
#修改docker的服务启动文件指定子网段
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
[Service]
Type=notify
#the default is not to use systemd for cgroups because the delegate issues still
#exists and systemd currently does not support the cgroup feature set required
#for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// -- containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost ~]# cat /run/flannel/subnet.env       #查看flannel为该节点分配的子网段
DOCKER_OPT_BIP="--bip=172.17.42.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
##说明：bip指定启动时的子网
DOCKER_NETWORK_OPTIONS=" --bip=172.17.42.1/24 -- ip-masq=false -- mtu=1450"
#重启docker服务

[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart docker

#查看flannel网络
[root@localhost ~]# ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.42.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::fc7c:e1ff:fe1d:224 prefixlen 64 scopeid 0x20<link>
ether fe:7c:e1:1d:02:24 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

#在两个node节点中都运行一个容器，测试能否ping通对方，能ping通则flannel部署成功
[root@localhost ~]# docker run -it centos:7 /bin/bash
[root@5f9a65565b53 /]# yum install net-tools -y
[root@5f9a65565b53 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
      inet 172.17.42.2 netmask 255.255.255.0 broadcast 172.17.84.255
      ether 02:42:ac:11:54:02 txqueuelen 0 (Ethernet)
 ....省略内容

4.部署master节点

#在 master上操作，api-server生成证书
上传master.zip到/root/k8s目录下
[root@localhost k8s]# unzip master.zip
[root@localhost k8s]# mkdir /opt/kubernetes/{ cfg,bin,ssl} -p
[root@localhost k8s]# mkdir k8s-cert
[root@localhost k8s]# cd k8s-cert/
[root@localhost k8s-cert]# ls      #上传到/root/k8s/k8s-cert/目录下
k8s-cert.sh     #利用脚本生成k8s证书
#脚本内容如下
cat > ca-config.json <<EOF          #ca证书配置文件
{ 
  "signing": { 
    "default": { 
      "expiry": "87600h"
    },
    "profiles": { 
      "kubernetes": { 
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF      #ca证书签名文件
{ 
    "CN": "kubernetes",
    "key": { 
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        { 
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
      	    "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -      
                                     #生成ca.pem和ca-key.pem（CA认证机构）
#-----------------------
cat > server-csr.json <<EOF
{ 
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "14.0.0.50",       #master01
      "14.0.0.80",       #master02
      "14.0.0.88",       #VIP
      "14.0.0.90",       #反向代理（master）
      "14.0.0.100",     #反向代理（backup）
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": { 
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        { 
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server                   
                   #生成kube-apiserver的tls认证证书和认证私钥server.pem和server-key.pem
#-----------------------
cat > admin-csr.json <<EOF
{ 
  "CN": "admin",
  "hosts": [],
  "key": { 
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    { 
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin          
     #kubectl的TLS认证证书和认证私钥，具有admin权限，admin.pem和admin-key.pem     
#-----------------------
cat > kube-proxy-csr.json <<EOF
{ 
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": { 
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    { 
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy   
                                       #生成kube-proxy-key.pem和kube-proxy.pem
#生成证书如下
[root@localhost k8s-cert]# ls *pem
admin-key.pem  ca-key.pem      kube-proxy-key.pem          server-key.pem
admin.pem        ca.pem              kube-proxy.pem                server.pem
#将ca证书和kube-apiserver证书复制到/opt/kubernetes/ssl/目录下
[root@localhost k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/
[root@localhost k8s-cert]# cd ..

#解压 kubernetes压缩包
[root@localhost k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s]# cd /root/k8s/kubernetes/server/bin
#复制软件包中bin目录下的关键命令文件
[root@localhost bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
[root@localhost k8s]# cd /root/k8s

#使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 可以随机生成序列号
[root@localhost k8s]# vim /opt/kubernetes/cfg/token.csv
0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
填写内容：序列号， 用户名，uid，用户组
（该文件为一个用户的描述文件，基本格式为 Token,用户名,UID,用户组；这个文件在 apiserver 启动时被 apiserver 加载，然后就相当于在集群内创建了一个这个用户；接下来就可以用 RBAC 给他授权。）

#二进制文件，token，证书都准备好，开启 apiserver
[root@localhost k8s]# bash apiserver.sh 14.0.0.50 https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
#检查进程是否启动成功
[root@localhost k8s]# ps aux | grep kube

#监听的https端口
[root@localhost k8s]# netstat -ntap | grep 6443
tcp 0 0 192.168.195.149:6443 0.0.0.0:* LISTEN 46459/kube-apiserve
tcp 0 0 192.168.195.149:6443 192.168.195.149:36806 ESTABLISHED 46459/kube-apiserve
tcp 0 0 192.168.195.149:36806 192.168.195.149:6443 ESTABLISHED 46459/kube-apiserve

#监听的http端口
[root@localhost k8s]# netstat -ntap | grep 8080
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN
46459/kube-apiserve

#启动scheduler服务
[root@localhost k8s]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@localhost k8s]# ps aux | grep ku
[root@localhost k8s]# chmod +x controller-manager.sh

#启动controller-manager服务
[root@localhost k8s]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

#查看master节点状态
[root@localhost k8s]# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy { "health":"true"}
etcd-1 Healthy { "health":"true"}
etcd-0 Healthy { "health":"true"}

5.部署node节点

#把master01节点上的kubelet、kube-proxy拷贝到node节点上去

[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.60:/opt/kubernetes/bin/
[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.70:/opt/kubernetes/bin/

#nod01节点操作（上传node.zip到/root目录下）
[root@localhost ~]# ls
anaconda-ks.cfg flannel-v0.10.0-linux-amd64.tar.gz  node.zip  公共 视频 文档 音乐
flannel.sh initial-setup-ks.cfg README.md 模板 图片 下载 桌 面
//解压 node.zip，获得 kubelet.sh；proxy.sh脚本
[root@localhost ~]# unzip node.zip

#在master上操作
[root@localhost k8s]# mkdir kubeconfig
[root@localhost k8s]# cd kubeconfig/
//将kubeconfig.sh文件进行重命名
[root@localhost kubeconfig]# mv kubeconfig.sh kubeconfig
//获取tokenID信息
[root@localhost ~]# cat /opt/kubernetes/cfg/token.csv
6351d652249951f79c33acdab329e4c4,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@localhost kubeconfig]# vim kubeconfig
#修改tokenID
kubectl config set-credentials kubelet-bootstrap \
-- token=6351d652249951f79c33acdab329e4c4 \
-- kubeconfig=bootstrap.kubeconfig
//设置环境变量 （可以写入到 /etc/profile中）
[root@localhost kubeconfig]# echo export PATH=$PATH:/opt/kubernetes/bin/ >> /etc/profile
[root@localhost kubeconfig]# source /etc/profile
[root@localhost kubeconfig]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy { "health":"true"}
etcd-0 Healthy { "health":"true"}
etcd-2 Healthy { "health":"true"}
//生成配置文件
[root@localhost kubeconfig]# bash kubeconfig 14.0.0.60 /root/k8s/k8s-cert/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
[root@localhost kubeconfig]# ls
bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig

//拷贝配置文件到 拷贝配置文件到 node节点
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.60:/opt/kubernetes/cfg/
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.70:/opt/kubernetes/cfg/

//首次启动时kubelet如何连接apiserver？
使用预设用户kubelet-bootstrap，但是需要群集角色绑定，将预设用户 kubelet-bootstrap与内置的 ClusterRole system:node-bootstrapper 绑定到一起，才能将权限用于连接 apiserver请求

[root@localhost kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

#在node01节点上操作
[root@localhost ~]# bash kubelet.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

//检查kubelet服务启动
[root@localhost ~]# ps aux | grep kube
root 106845 1.4 1.1 371744 44780 ? Ssl 00:34 0:01 /opt/kubernetes/bin/kubelet -- logtostderr=true -- v=4 -- hostname-override=192.168.195.150 -- kubeconfig=/opt/kubernetes/cfg/kubelet.kubconfig -- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig -- config=/opt/kubernetes/cfgkubelet.config -- cert-dir=/opt/kubernetes/ssl -- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 106876 0.0 0.0 112676 984 pts/0 S+ 00:35 0:00 grep -- color=auto kube
#master01上操作（手动签发）
//检查node01节点的请求
[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 4m27s kubelet-bootstrap Pending（等待集群给该节点颁发证书）

[root@localhost kubeconfig]# kubectl certificate approve
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A
certificatesigningrequest.certificates.k8s.io/node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A approved  

[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 8m56s kubelet-bootstrap Approved,Issued（Approved代表已经被允许加入群集）

//查看群集节点，成功加入 node01节点
[root@localhost kubeconfig]# kubectl get node
NAME STATUS ROLES AGE VERSION
14.0.0.50 Ready <none> 118s v1.12.3

#在 node01节点操作，启动 proxy服务
[root@localhost ~]# bash proxy.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@localhost ~]# systemctl status kube-proxy.service
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2020-02-02 00:47:29 CST; 11s ago
Main PID: 108006 (kube-proxy)
Memory: 7.5M
CGroup: /system.slice/kube-proxy.service
‣ 108006 /opt/kubernetes/bin/kube-proxy -- logtostderr=true -- v=4 -- hostname-override=1...

node02节点部署

#在node01节点操作
//把现成的 /opt/kubernetes目录复制到其他节点进行修改即可
[root@localhost ~]# scp -r /opt/kubernetes/ root@14.0.0.70:/opt/
//把 kubelet，kube-proxy的 service文件拷贝到 node2中
[root@localhost ~]# scp /usr/lib/systemd/system/{ kubelet,kube-proxy}.service root@14.0.0.70:/usr/lib/systemd/system/

#在node02节点上操作，进行修改
//首先删除复制过来的证书，等会 node02会自行申请证书
[root@localhost ~]# cd /opt/kubernetes/ssl/
[root@localhost ssl]# rm -rf *
//修改配置文件kubelet kubelet.config kube-proxy（三个配置文件）
[root@localhost ssl]# cd ../cfg/
[root@localhost cfg]# vim kubelet
KUBELET_OPTS="-- logtostderr=true \ -- v=4 \ -- hostname-override=14.0.0.70 \ -- kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ -- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ -- config=/opt/kubernetes/cfg/kubelet.config \ -- cert-dir=/opt/kubernetes/ssl \ -- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
[root@localhost cfg]# vim kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 14.0.0.70
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
~
[root@localhost cfg]# vim kube-proxy
KUBE_PROXY_OPTS="-- logtostderr=true \ -- v=4 \ -- hostname-override=14.0.0.70 \ -- cluster-cidr=10.0.0.0/24 \ -- proxy-mode=ipvs \ -- kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
//启动服务
[root@localhost cfg]# systemctl start kubelet.service
[root@localhost cfg]# systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@localhost cfg]# systemctl start kube-proxy.service
[root@localhost cfg]# systemctl enable kube-proxy.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

//在master上操作查看请求
[root@localhost k8s]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU 15s kubelet-bootstrap Pending

//授权许可加入群集
[root@localhost k8s]# kubectl certificate approve node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU
certificatesigningrequest.certificates.k8s.io/node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU approved

//查看群集中的节点
[root@localhost k8s]# kubectl get node  #两个node节点均显示Ready代表都已成功加入
NAME STATUS ROLES AGE VERSION
14.0.0.60 Ready <none> 21h v1.12.3
14.0.0.70 Ready <none> 37s v1.12.3

实验故障及如何处理

如果实验最后查看群集中的节点时，发现有node处于noready状态，如何处理？

1.先检查网络，查看各个节点间网络是否互通；
2.网络没问题再检查kubelet，检查kubeconfig中的三个配置（apiserver、tokenID、证书）是否正确；
3.使用公网上的主机访问node中的业务，如果内外网都无法访问服务器说明该node服务器可能已经宕机了，需要联系现场运维工程师查看。