正文

第三篇（二进制部署k8s集群---flannel网络和keepalived+haproxy高可用）(代码片段)

author  author  2022-12-12  166

关键词：

此文将搭建flannel网络，目的使跨主机的docker能够互相通信，也是保障kubernetes集群的网络基础和保障，和ha高可用。
部署的服务器为：
master1 192.168.206.31
master2 192.168.206.32
master3 192.168.206.33
node1 192.168.206.41
node2 192.168.206.42
node3 192.168.206.43
VIP：192.168.206.30
ha1 192.168.206.36
ha2 192.168.206.37

一、生成Flannel网络TLS证书

在所有集群节点都安装Flannel，下面的操作在k8s-master1上进行。
1、创建证书签名请求

cat > flanneld-csr.json <<EOF

  "CN": "flanneld",
  "hosts": [],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "ST": "Zhejiang",
      "L": "hangzhou",
      "O": "k8s",
      "OU": "System"
    
  ]

EOF

该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；

2、生成证书和私钥：

cfssl gencert -ca=/data/ssl/ca.pem   -ca-key=/data/ssl/ca-key.pem   -config=/data/ssl/ca-config.json   -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

创建证书存放目录：
 mkdir /opt/kubernetes/ssl/flannel

这里是复制到3master+3node上
cp flanneld*.pem /opt/kubernetes/ssl/flannel

二、部署 Flannel
1、下载安装Flannel

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz
cp flanneld,mk-docker-opts.sh /opt/kubernetes/bin/

2、向 etcd 写入网段信息
下面2条命令在etcd集群中任意一台执行一次即可，也是是创建一个flannel网段供docker分配使用

etcdctl --ca-file=/opt/kubernetes/ssl/etcd/ca.pem --cert-file=/opt/kubernetes/ssl/etcd/etcd.pem --key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem mkdir /opt/kubernetes/network
etcdctl --ca-file=/opt/kubernetes/ssl/etcd/ca.pem --cert-file=/opt/kubernetes/ssl/etcd/etcd.pem --key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem mk /opt/kubernetes/network/config ‘"Network":"172.30.0.0/16","SubnetLen":24,"Backend":"Type":"vxlan"‘

3、创建system unit文件

cat > /etc/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/kubernetes/bin/flanneld   -etcd-cafile=/opt/kubernetes/ssl/flannel/ca.pem   -etcd-certfile=/opt/kubernetes/ssl/flannel/flanneld.pem   -etcd-keyfile=/opt/kubernetes/ssl/flannel/flanneld-key.pem   -etcd-endpoints=https://192.168.206.31:2379,https://192.168.206.32:2379,https://192.168.206.33:2379   -etcd-prefix=/opt/kubernetes/network
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入到 /run/flannel/docker 文件中，后续 docker 启动时使用这个文件中参数值设置 docker0 网桥。
flanneld 使用系统缺省路由所在的接口和其它节点通信，对于有多个网络接口的机器（如，内网和公网），可以用 -iface=enpxx 选项值指定通信接口。

4、启动flannel并且设置开机自启动

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

5、查看flannel分配的子网信息

[root@k8s-master1 ~]# cat /run/flannel/docker 
DOCKER_OPT_BIP="--bip=172.30.94.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.30.94.1/24 --ip-masq=true --mtu=1450"

[root@k8s-master1 ~]# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=172.30.0.0/16
FLANNEL_SUBNET=172.30.94.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

/run/flannel/docker是flannel分配给docker的子网信息，/run/flannel/subnet.env包含了flannel整个大网段以及在此节点上的子网段。

6、查看flannel网络是否生效

Last login: Thu Nov 19 09:28:40 2020 from 192.168.206.1
[root@k8s-master1 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6e:7f:49 brd ff:ff:ff:ff:ff:ff
    inet 192.168.206.31/24 brd 192.168.206.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bbd4:6d75:22b1:e631/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::129b:129d:71ca:5d94/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::1b37:c32:6cc4:be75/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether de:b1:04:6f:d6:57 brd ff:ff:ff:ff:ff:ff
    inet 172.30.65.0/32 brd 172.30.65.0 scope global flannel.1
       valid_lft forever preferred_lft forever

三、安装docker、配置docker支持flannel网络
1、所有node安装docker

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

#安装指定版本，这里安装18.06
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-18.06.1.ce-3.el7

systemctl start docker && systemctl enable docker

2、配置docker支持flannel网络，所有docker节点都操作

[root@k8s-master1 ~]# vi /etc/systemd/system/multi-user.target.wants/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

3、重启docker，使配置生效

systemctl daemon-reload
systemctl restart docker

4、查看所有集群主机的网络情况

etcdctl --ca-file=/opt/kubernetes/ssl/etcd/ca.pem --cert-file=/opt/kubernetes/ssl/etcd/etcd.pem --key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem ls /opt/kubernetes/network/subnets

四、keepalived+haproxy高可用部署。
部署服务器
ha1 192.168.206.36
ha2 192.168.206.37
1、所有haproxy安装haproxy

yum install -y haproxy

cat <<EOF > /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
defaults
    mode                    tcp
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m
frontend  k8s-api
    bind        *:6443
    bind        *:443
    mode        tcp
    option      tcplog
    default_backend k8s-api
backend k8s-api
    mode        tcp
    option      tcplog
    option      tcp-check
    balance     roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server master1 192.168.206.31:6443 check
    server master2 192.168.206.32:6443 check
    server master3 192.168.206.33:6443 check
EOF

2、启动所有haproxy

systemctl start haproxy
systemctl status haproxy
systemctl enable haproxy

3、所有haproxy安装keepalived

yum install -y keepalived

cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs 
   router_id LVS_K8S


vrrp_instance VI_1 
    state MASTER(BACKUP)
    interface ens33
    virtual_router_id 51
    priority 100(备50)
    advert_int 1
    authentication 
        auth_type PASS
        auth_pass 1111
    
    virtual_ipaddress 
        VIP/24
    

EOF

4、所有haproxy启动keepalived

systemctl restart keepalived
systemctl status keepalived
systemctl enable keepalived

启动完成可用查看vip，或关闭主ha看vip是否偏移。

kubernetes二进制部署（单master节点）（理论部分，详细部署步骤在下一篇哦~）(代码片段)

Kubernetes二进制部署（单Master节点）（理论部分，详细部署步骤在下一篇哦~）Kubernetes二进制部署一、环境准备二、部署etcd集群1、准备签发证书环境2、签发证书步骤3、etcd-cert.sh和etcd.sh脚本4、部署流程5、使用... 查看详情

kubernetes集群之二进制安装部署（单master节点）(代码片段)

目录前言一、常见的K8S按照部署方式二、安装部署分析三、首先部署ETCD集群3.1ETCD介绍3.2准备CFSSL证书签发环境3.3环境部署3.3.1搭建ETCD步骤3.3.2下载准备CFSSL证书制作工具3.3.3上传etcd-cert.sh和etcd.sh到/opt/k8s/目录中3.3.4安装ETCD服务3.3.... 查看详情

二进制部署k8s单节点(代码片段)

Kubernetes集群部署环境规划一、ETCD数据库集群部署1.安装cfssl证书生成工具2.生成证书3.etcd部署3.1创建对应目录，拷贝相应文件3.2查看启动脚本3.3执行脚本等待节点加入3.4将文件拷贝到其他节点，并修改配置文件3.5启动etcd&#... 查看详情

kubernetes节点服务搭建————二进制部署|单master节点配置（一)(etcd和flannel)(代码片段)

文章目录常见的K8s按照部署方式Kubernetes二进制部署部署etcd集群master01上操作部署在node节点上修改配置部署Docker引擎flannel网络配置常见的通信方式flannel的工作流程flannel的搭建部署总结常见的K8s按照部署方式MinikubeMinikube是一个工... 查看详情

kubernetes节点服务搭建————二进制部署|单master节点配置（一)(etcd和flannel)(代码片段)

k8s集群架构的二进制部署——k8s集群学习的基础(代码片段)

K8S集群架构的二进制部署一、部署etcd二、部署docker引擎三、flannel网络配置四、部署master组件五、node节点部署一、部署etcdMaster上操作[root@192etcd-cert]#cd/usr/local/bin/[root@192bin]#rz-E（cfsslcfssl-certinfocfssljson）[root 查看详情

k8s------kubernetes单master集群二进制搭建(代码片段)

目录前言Kubernetes单Master节点集群二进制部署一.Kubernetes单Master集群架构二.环境准备三.Etcd集群部署1.准备签发证书环境2.生成CA证书、etcd服务器证书以及私钥3.安装Etcd4.在Master01节点启动Etcd集群节点015.将证书，命令文件，... 查看详情

k8s单节点集群二进制部署（步骤详细，图文详解）(代码片段)

k8s单节点集群二进制部署（步骤详细，图文详解）一、k8s集群搭建环境准备1、etcd集群master01node01node02所有node节点部署docker引擎2、flannel网络插件3、搭建master组件4、搭建node组件（1）node1节点（2）node2节... 查看详情

k8s单节点集群二进制部署（步骤详细，图文详解）(代码片段)

k8s—二进制部署安装(代码片段)

安装步骤一、准备工作二、部署etcd集群master节点node节点（1/2）查看集群状态三、部署docker引擎node节点（1/2）四、设置flannel网络master节点node节点操作（1/2）五、部署master组件六、node节点部署master节点操作... 查看详情

kubernetes二进制部署——flannel网络(代码片段)

Kubernetes二进制部署——Flannel网络一、Flannel简介二、Flannel原理三、Flannel的作用四、Flannel网络配置1.node节点安装docker2.master配置1、写入分配的子网段到ETCD中，供flannel使用(master主机）3、node节点操作1、上传flannel软件包到... 查看详情

kubernetes二进制部署——flannel网络(代码片段)

云原生kubernetes系列第三篇二进制部署单节点kubernetes（k8s）v1.20（不要因为别人都在交卷，自己就乱写答案）(代码片段)

文章目录系列文章目录前言一、实验环境二、操作系统初始化配置三、部署etcd集群3.1准备签发证书环境3.2准备cfssl证书生成工具3.3生成Etcd证书3.4启动etcd服务3.5检查群集状态四、部署docker引擎五、部署Master组件5.1准备软件包5.2创... 查看详情

k8s集群日志

硬件环境：三台虚拟机，10.10.20.203部署docker、etcd、flannel、kube-apiserver、kube-controller-manager、kube-scheduler、elsticsearch、kibana10.10.20.206 部署docker、flannel、kubelet、kube-proxy、filebeat10.10.20.207部署docker 查看详情

k8s二进制部署之node安装docker(代码片段)

前一篇：k8s二进制部署之部署Etcd集群后一篇：k8s二进制部署之网络部署（Flannel）介绍本篇展示docker离线部署模式，应为有些时候公司内网环境时不允许访问互联网的，这种情况只能使用离线模式。当然... 查看详情

k8s集群容器监控

硬件环境：两台虚拟机，10.10.20.203部署docker、etcd、flannel、kube-apiserver、kube-controller-manager、kube-scheduler、influxdb、grafana10.10.20.206 部署docker、flannel、kubelet、kube-proxy、cadvisor10.10.20.207部署docker、fl 查看详情

flannel网络组件部署(代码片段)

...实现集群的网络互联互通。目前可选的组件比较多，例如flannel、calico、weave等，各容器网络组件对比可参考文档：http://dockone.io/article/2599本文介绍flannel网络组件的部署，配置环境在完成前文etcd集群和tls认证配置后。一、生成flan... 查看详情