Kubeadm安装

大家好，欢迎来到IT知识分享网。

VM虚拟机两台centos7
4核8g

安装步骤：

一、环境准备(所有节点)

1. centos内核升级（非必须，但建议升级）

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

查看可升级的内核版本

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available

安装内核

yum --enablerepo=elrepo-kernel install -y kernel-lt

查看系统内可用内核

awk -F\' '$1=="menuentry " { 
   print i++ " : " $2}' /etc/grub2.cfg

设置开机从新内核版本启动（其中 0 是上面查询出来的可用内核编号）

grub2-set-default 0

reboot

2. 配置静态ip

vim /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=6bcaf60a-b6f8-43de-a915-460a9a674532
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.32.135
GATEWAY=192.168.32.2
DNS1=192.168.32.2

systemctl restart network

3. 设置主机名及hosts文件相互解析

hostnamectl set-hostname master
vi /etc/hosts
192.168.32.136     master
192.168.32.136     node1

4. 安装依赖包

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git 

5. 关闭防火墙、设置iptables并设置空规则

systemctl stop firewalld && systemctl disable firewalld
yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save

6. 禁用swap交换分区、关闭SELINUX

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

通过运行setenforce 0并sed …有效禁用SELinux，将其设置为许可模式。这是允许容器访问主机文件系统所必需的，例如Pod网络所需要的。您必须执行此操作，直到kubelet中的SELinux支持得到改善。
7. 调整内核参数，对于k8s

cat > kubernetes.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 #开启网桥模式(必须) net.bridge.bridge-nf-call-iptables = 1 #开启网桥模式(必须) net.ipv4.fp_forward = 1 net.ipv4.tcp_tw_recycle = 0 vm.swappiness = 0 vm.overcommit_memory = 1 #不检测物理内存是否够用 net.ipv6.conf.all.disable_ipv6 = 1 #关闭ipv6协议(必须) net.netfilter.nf_conntrack_max = 2310720 EOF

cp kubernetes.conf  /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

8. 关闭系统不需要的服务

systemctl stop postfix && systemctl disable postfix

9. 设置日志的保存方式(默认使用rsyslogd和systemd journald ，设置默认使用journald)

mkdir /var/log/journal                 #持久化保存日志的目录
mkdir /etc/systemd/journal.conf.d      #日志配置文件目录
cat > /ect/systemd/journal.conf.d/99-prophet.conf  << EOF [Journal] Storage = persistent #持久化保存到磁盘 Compress = yes #压缩历史日志 SyncIntervalSec = 5m RateLimitInterval = 30s RateLimitBurst = 1000 SystemMaxUse = 10G #最大占用空间 SystemMaxFileSize = 200M #单文件最大保存200m MaxRetentionSec = 2week #日志保存时间两周 ForwardToSyslog = no #不将日志转发到syslog EOF

systemctl restart systemd-journald

10. 开启kube-proxy的ipvs前置条件

modprobe br_netfilter         #加载netfilter模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

二、安装docker-ce

1. 安装必要的工具包

yum install yum-utils device-mapper-persistent-data lvm2

2. 导入阿里云的docker镜像仓库

yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

3. 安装最新docker-ce

yum update -y && yum install docker-ce

4. 创建daemon.json配置文件

mkdir /etc/docker

cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" } } EOF

创建目录存放docker生成的子配置文件

mkdir -p /etc/systemd/system/docker.service.d

6. 重启docker

systemctl daemon-reload &&
systemctl restart docker &&
systemctl enable docker

三、安装 kubeadm, kubelet 和 kubectl

1. 配置kubernetes阿里源（所有节点）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2. 安装 kubeadm, kubelet 和 kubectl（所有节点）

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable kubelet

kubectl开机自启动是kubeadm的强制要求

四、kubeadm初始化集群

1. pull Kubeadm必须的镜像
   使用kubeadm的子命令列出需要的镜像列表

kubeadm config images list

因为网络安全的原因直接pull会失败

# 导出配置文件
kubeadm config print init-defaults --kubeconfig ClusterConfiguration > kubeadm.yml

修改kubeadm.yml文件下述注释部分

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  # 修改为主节点 IP
  advertiseAddress: 192.168.141.130
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: kubernetes-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: { 
   }
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
# 国内不能访问 Google，修改为阿里云
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
# 修改版本号
kubernetesVersion: v1.16.3
networking:
  dnsDomain: cluster.local
  # 配置成 flannel 的默认网段
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: { 
   }
# 将默认调度方式改为ipvs
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

# 查看所需镜像列表
kubeadm config images list --config kubeadm.yml
# 拉取镜像
kubeadm config images pull --config kubeadm.yml

2. 初始化kubeadm

kubeadm init --config kubeadm.yml

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.32.136:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ccaabd30fa3f25b5d696adf09d826709f7029f8902d2c08ac98a9af76c8acddb 

说明初始化成功；备份kubeadm join ······用于node节点加入集群管理
3. 初始化成功以后
如果你使用root用户：

export KUBECONFIG=/etc/kubernetes/admin.conf

非root用户：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

五、master节点安装pod网络插件（flannel）

两种安装方式：

1. 在线安装fannel网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

2. 使用kubectl

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl create -f kube-flannel.yml

kubectl get pod -n kube-system

六、node节点加入

node节点使用kubeadm init成功后输出的kubeadm join指令
如果token忘记：

kubeadm token list

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

将上述命令生成的值合为如下命令：

kubeadm join 192.168.18.64:6443 --token TOKEN --discover-token-ca-cert-hash sha256:HASH

验证

kubectl get cs 

输出：

NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   { 
   "health": "true"}

kubectl get node

输出：

NAME      STATUS    ROLES     AGE       VERSION
worker3   Ready     master    1d        v1.10.0

创建一个两个副本的Nginx服务

kubectl run nginx --image=nginx --replicas=2
kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort

查看服务

kubectl get service

通过IP+port访问k8s创建的nginx服务了

安装中遇到的一些问题

1.kubeadm初始化是kube-proxy一直处于CrashLoopBackOff状态
排查过程：
查看kube-proxy模块日志

kubectl logs  kube-proxy-6lp79 --namespace=kube-system

日志如下：

failed validate: KubeProxyConfiguration.Mode.ProxyMode: Invalid value: "ipv": must be iptables,ipvs,userspace or blank (blank means the best-available proxy [currently iptables])

不难发现ProxyMode值为ipv，是无效的
解决方案：更改kube-proxy配置

kubectl edit configmap kube-proxy -n kube-system

minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: "ipv"                          # 把ipv改成ipvs
    nodePortAddresses: null

kubernetes资源清单常用字段说明