k8s1.23.15集群二进制部署

一、前言

    二进制部署1.23.15版本k8s集群，etcd集群部署与k8s集群节点复用，手动颁发集群证书

    主机信息如下

二、部署

   集群所有节点主机基础优化

   1.关闭防火墙

    systemctl stop firewalld && systemctl disable firewalld

   2.关闭selinux服务

setenforce 0  #临时关闭

sed -i 's/enforcing/disabled/' /etc/selinux/config #永久关闭

3.关闭系统交换分区

swapoff -a #临时关闭

sed -ri 's/.*swap.*/#&/' /etc/fstab  #永久关闭

4.设置主机名称(三台主机名称不一样按照表格上的主机名填)

hostnamectl set-hostname k8s-master01

5.编辑hosts文件

cat > /etc/hosts << EOF

10.1.60.125 k8s-master01

10.1.60.126 k8s-node01

10.1.60.127 k8s-node02

10.1.60.128 k8s-node03

EOF

6.配置时间同步

yum -y install chrony

systemctl strat chronyd && systemctl enbale chronyd

7.配置ipvs

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack_ipv4

EOF

#授权并生效ipvs配置

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules

yum install -y ipset ipvsadm

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

 sudo modprobe overlay && sudo modprobe br_netfilter

8.将桥接的IPv4流量传递到iptables

cat > /etc/sysctl.d/k8s.conf << EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

EOF

sysctl --system  #配置生效

9.配置所有主机互相免密登录

ssh-keygen

ssh-copy-id -i ~/.ssh/id_rsa.pub 10.1.60.126   #将公钥拷贝到126主机上，实现免密登录

ssh-copy-id -i ~/.ssh/id_rsa.pub 10.1.60.127

ssh-copy-id -i ~/.ssh/id_rsa.pub 10.1.60.128

部署k8s集群服务 

安装证书生成工具cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

创建证书存储目录

mkdir -p ~/TLS/{etcd,k8s}
cd TLS/etcd

创建证书文件

vi ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

vi ca-csr.json

{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShenZhen",
            "ST": "ShenZhen"
        }
    ]
}

生成根证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

查看目录下是否生成ca-key.pem和ca.pem这两个根证书

ls *.pem

创建etcd证书申请文件

vi server-csr.json

{
    "CN": "etcd",
    "hosts": [
    "10.1.60.125",    #填写etcd集群所有节点的ip地址
    "10.1.60.126",
    "10.1.60.127"
   
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C:": "CN",
            "L": "ShenZhen",
            "ST": "ShenZhen"
        }
    ]
}

 生成etcd证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

查看目录下是否生成server-key.pem server.pem这两个证书文件

ls *.pem

部署etcd集群

 查看k8s集群版本对应版本的etcd版本，我这里使用的是3.5.1版本的etcd

安装包下载参考：https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz

创建etcd工作目录并解压安装包 

cd ~
mkdir -p /opt/etcd/{bin,cfg,ssl} 
wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz
mv etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

创建etcd配置文件

vi /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd-1"   #etcd集群中的节点名称
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"   #etcd数据目录
ETCD_LISTEN_PEER_URLS="https://10.1.60.125:2380"      #etcd节点通讯地址端口
ETCD_LISTEN_CLIENT_URLS="https://10.1.60.125:2379"    #etcd客户端访问地址端口
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.60.125:2380"  #集群通讯地址端口
ETCD_ADVERTISE_CLIENT_URLS="https://10.1.60.125:2379"       #集群客户端通讯地址端口
ETCD_INITIAL_CLUSTER="etcd-1=https://10.1.60.125:2380,etcd-2=https://10.1.60.126:2380,etcd-3=https://10.1.60.127:2380"        #etcd集群所有节点地址和端口
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"   #集群之间通讯的token
ETCD_INITIAL_CLUSTER_STATE="new"          #加入etcd集群的状态，new是新集群，existing是加入已有集群

使用systemd管理etcd服务

vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf    #指定配置文件
ExecStart=/opt/etcd/bin/etcd \
    --cert-file=/opt/etcd/ssl/server.pem \     #指定证书
    --key-file=/opt/etcd/ssl/server-key.pem \
    --peer-cert-file=/opt/etcd/ssl/server.pem \
    --peer-key-file=/opt/etcd/ssl/server-key.pem \
    --trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

拷贝生成的etcd证书到etcd工作目录下

cp ~/TLS/etcd/ca*pem /opt/etcd/ssl/
cp ~/TLS/etcd/server*pem /opt/etcd/ssl/

将节点1上生成的关于etcd的所有文件拷贝到其余两个etcd节点上

#拷贝到etcd节点二上
scp -r /opt/etcd/ root@10.1.60.126:/opt/
scp /usr/lib/systemd/system/etcd.service root@10.1.60.126:/usr/lib/systemd/system/
#拷贝到etcd节点三上
scp -r /opt/etcd/ root@10.1.60.127:/opt/
scp /usr/lib/systemd/system/etcd.service root@10.1.60.127:/usr/lib/systemd/system/

拷贝完成后编辑两个节点的etcd配置文件

vi /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd-2"        #更改此项为不同的节点名称
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.1.60.126:2380"     #更改此项为该节点对应的ip地址
ETCD_LISTEN_CLIENT_URLS="https://10.1.60.126:2379"    #更改此项为该节点对应的ip地址
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.60.126:2380"  #更改此项为该节点对应的ip地址
ETCD_ADVERTISE_CLIENT_URLS="https://10.1.60.126:2379"    #更改此项为该节点对应的ip地址
ETCD_INITIAL_CLUSTER="etcd-1=https://10.1.60.125:2380,etcd-2=https://10.1.60.126:2380,etcd-3=https://10.1.60.127:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" 
ETCD_INITIAL_CLUSTER_STATE="new"

etcd集群所有节点启动etcd服务

systemctl daemon-reload

systemctl start etcd && systemctl enable etcd

etcd集群所有节点服务启动后查看状态是否正常

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.1.60.125:2379,https://10.1.60.126:2379,https://10.1.60.127:2379" endpoint health

输出以下信息表面etcd集群状态正常

k8s集群所有节点 安装docker服务

docker版本也需要对应k8s版本，这里选用19.03.9版本

下载参考：https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

下载安装包并解压 

cd ~
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
tar -zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

 使用systemd管理docker服务

vi /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

配置docker镜像加速

mkdir /etc/docker

vi /etc/docker/daemon.json

{
  "registry-mirrors": ["https://62d7sxch.mirror.aliyuncs.com"]   #配置阿里云镜像加速地址
}

启动docker服务

systemctl daemon-reload

systemctl start docker && systemctl enable docker

创建k8s集群自签证书文件

cd ~/TLS/k8s

vi ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

vi ca-csr.json

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShenZhen",
            "ST": "ShenZhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

生成根证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

查看是否生成ca-key.pem和ca.pem这两个根证书

ls *.pem

创建kube-apiserver证书申请文件

vi server-csr.json

{
    "CN": "kubernetes",
    "hosts": [
      "20.0.0.1",        #定义k8s集群地址，若是一个局域网中有多个k8s集群，k8s集群地址的网段不能相同，否则会冲突
      "127.0.0.1",
      "10.1.60.125",    #填写k8s集群节点地址
      "10.1.60.126",
      "10.1.60.127",
      "10.1.60.128",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShenZhen",
            "ST": "ShenZhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

 生成kube-apiserver证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

查看生成的证书

ls server*pem

创建kube-proxy证书申请文件

vi kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "ShenZhen",
      "ST": "ShenZhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

 生成kube-proxy证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

查看生成的证书

ls kube-proxy*pem

部署master节点所需服务

 在gitlab中下载1.23.10版本的k8s server安装包

参考：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#downloads-for-v12310

下载并解压k8s server安装包

cd ~
wget https://dl.k8s.io/v1.23.10/kubernetes-server-linux-amd64.tar.gz
tar -zxvf kubernetes-server-linux-amd64.tar.gz

创建k8s工作目录

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

部署kube-apiserver服务

创建kube-apiserver配置文件

vi /opt/kubernetes/cfg/kube-apiserver.conf

KUBE_APISERVER_OPTS="--allow-privileged=true \   #启用授权
--feature-gates=RemoveSelfLink=false \   #K8S1.20后移除了对SelfLink的支持，而NFS需要SelfLink的支持
--etcd-servers=https://10.1.60.125:2379,https://10.1.60.126:2379,https://10.1.60.127:2379 \     #配置etcd集群所有节点地址和客户端端口
--bind-address=10.1.60.125 \    #apiserver服务监听地址
--secure-port=6443 \          #apiserver服务监听端口
--advertise-address=10.1.60.125 \     #k8s集群通讯地址
--service-cluster-ip-range=20.0.0.0/24 \   #k8s集群网络地址段，不能和局域网中的其它k8s集群地址段相同
--enable-admission-plugins=NodeRestriction \     #准入控制插件
--authorization-mode=RBAC,Node \       #授权模式
--enable-bootstrap-token-auth=true \   #启用 bootstrap-tokenren认证，为了让node节点kubelet自由的颁发证书
--token-auth-file=/opt/kubernetes/cfg/token.csv \    #指定token文件
--service-node-port-range=30000-32767 \            #定义nodeport使用端口范围
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \   #指定apiserver访问kubelet客户端的证书
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \     #指定apiserver访问kubelet客户端的证书
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \        #指定apiserver https证书
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \       #指定apiserver https证书
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--service-account-issuer=api \
--service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \         #指定连接etcd集群证书
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \
--requestheader-allowed-names=kubernetes \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

 拷贝根证书和apiserver证书到k8s工作目录中

cp ~/TLS/k8s/ca*pem /opt/kubernetes/ssl/
cp ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

生成token文件所需的密钥

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

使用生成的密钥编辑token文件

vi /opt/kubernetes/cfg/token.csv

c5cdf6375ed70ce66c52de20eb350bb6,kubelet-bootstrap,10001,"system:node-bootstrapper"

使用systemd管理kube-apiserver服务

vi /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

 启动kube-apiserver服务

systemctl daemon-reload

systemctl start kube-apiserver && systemctl enable kube-apiserver

部署kube-controller-manager服务

编辑kube-controller-manager服务配置文件

vi /opt/kubernetes/cfg/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--leader-elect=true \
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=20.244.0.0/16 \    #指定k8s pod服务使用的网段
--service-cluster-ip-range=20.0.0.0/24 \    #指定k8s ervice服务使用的网段
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \    #指定kubelet颁发证书所需的根证书
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--cluster-signing-duration=87600h0m0s"   #指定kubelete颁发证书的有效期

创建controller-manager服务证书配置文件

cd ~/TLS/k8s

vi kube-controller-manager-csr.json

{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "ShenZhen", 
      "ST": "ShenZhen",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

查看生成的证书文件

ls kube-controller-manager*.pem

创建生成kubeconfig文件脚本

vi kube-controller-manager.sh

KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://10.1.60.125:6443"   #填写kube-apiserver服务地址和端口

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-controller-manager \
  --client-certificate=./kube-controller-manager.pem \
  --client-key=./kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

赋权执行脚本生成kubeconfig文件

chmod +x kube-controller-manager.sh

sh kube-controller-manager.sh

使用systemd管理kube-controller-manager服务

vi /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动kube-controller-manager服务

systemctl daemon-reload
systemctl start kube-controller-manager && systemctl enable kube-controller-manager

部署kube-scheduler服务

编辑kube-scheduler服务配置文件

vi /opt/kubernetes/cfg/kube-scheduler.conf

KUBE_SCHEDULER_OPTS="--leader-elect \
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \
--bind-address=127.0.0.1"

创建kube-scheduler服务证书配置文件

vi kube-scheduler-csr.json

{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "ShenZhen",
      "ST": "ShenZhen",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

查看生成的证书文件

ls kube-scheduler*.pem

 编辑生成kubeconfig配置文件脚本

vi kube-scheduler.sh

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://10.1.60.125:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-scheduler \
  --client-certificate=./kube-scheduler.pem \
  --client-key=./kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

赋权执行脚本生成kubeconfig文件

chmod +x kube-scheduler.sh
sh kube-scheduler.sh

使用systemd管理kube-scheduler服务

vi /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failurea

[Install]
WantedBy=multi-user.target

启动kube-scheduler服务

systemctl daemon-reload
systemctl start kube-scheduler && systemctl enable kube-scheduler

创建kubectl证书配置文件

vi admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "ShenZhen",
      "ST": "ShenZhen",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

查看生成的证书文件

ls admin*.pem

 编辑生成kubeconfig的配置文件脚本

vi kubectl.sh

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://10.1.60.125:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \
  --client-certificate=./admin.pem \
  --client-key=./admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建kubectl的kubeconfig存放目录

mkdir /root/.kube

赋权执行脚本生成kubeconfig文件

chmod +x kubectl.sh
sh kubectl.sh

使用kubectl命令查看master节点上的组件是否正常运行

kubectl get cs

 授权kubelet-bootstrap用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
#若想删除则使用以下命令
#kubectl delete clusterrolebinding kubelet-bootstrap

部署work节点所需服务

master节点也作为work节点

在master节点拷贝work节点所需的组件服务到k8s工作目录

cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin    #将work节点所需服务组件拷贝到k8s工作目录

部署kubelet服务

编辑kubelet配置文件

vi /opt/kubernetes/cfg/kubelet.conf

KUBELET_OPTS="--hostname-override=k8s-master01-test \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"   #管理Pod网络容器的镜像

创建kubelet定义参数文件

vi /opt/kubernetes/cfg/kubelet-config.yml

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 20.0.0.2        #定义dns服务的ip地址，后面部署coredns服务会使用到
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110

编辑生成kubeconfig的配置文件脚本

vi kubelet.sh

KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig"
KUBE_APISERVER="https://10.1.60.125:6443"
TOKEN="c5cdf6375ed70ce66c52de20eb350bb6"   #该token需要和上面生成的token.csv中的token一致

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

赋权并执行脚本生成kubeconfig文件

chmod +x kubelet.sh
sh kubelet.sh

使用systemd管理kublete服务

[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动kubelet服务

systemctl daemon-reload
systemctl start kubelet && systemctl enable kubelet

审批kubelet证书申请并加入集群

kubectl get csr   #将该命令输出的申请名称填入下面命令中审批
kubectl certificate approve node-csr-ldQlmnDLt-PFbwy2w5NIkwNB8eS4uHc3-hGvSu97elo
kubectl get node  #查看集群节点，就会发现有一个master节点

部署kube-proxy服务

编辑kube-proxy配置文件

vi /opt/kubernetes/cfg/kube-proxy.conf

KUBE_PROXY_OPTS="--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

创建kube-proxy定义参数文件

vi /opt/kubernetes/cfg/kube-proxy-config.yml

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master01-test
clusterCIDR: 20.244.0.0/16

创建kube-proxy生成证书的配置文件

vi kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "ShenZhen",
      "ST": "ShenZhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

查看生成的证书文件

ls kube-proxy*.pem

编辑生成kubeconfig的配置文件脚本

vi kube-proxy.sh

KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig"
KUBE_APISERVER="https://10.1.60.125:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

赋权并执行脚本生成kubeconfig文件

chmod +x kubelet.sh
sh kubelet.sh

使用systemd管理kube-proxy服务

vi /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动kube-proxy服务

systemctl daemon-reload
systemctl start kube-proxy  && systemctl enable kube-proxy

部署calico网络组件

参考：k8s集群使用calico网络组件_k8s集群与calico版本对应关系-CSDN博客

授权 apiserver服务访问kubelet

vi /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes

work节点加入集群

在其余work节点创建k8s工作目录

mkdir -p /usr/local/k8s/kubernetes/{bin,cfg,ssl,logs}

在master节点上拷贝证书以及组件服务到所有work节点上

scp -r /opt/kubernetes root@10.1.60.126:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@10.1.60.126:/usr/lib/systemd/system

scp -r /opt/kubernetes root@10.1.60.127:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@10.1.60.127:/usr/lib/systemd/system

scp -r /opt/kubernetes root@10.1.60.128:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@10.1.60.128:/usr/lib/systemd/system

 在拷贝文件后的所有work节点删除kubelet的证书文件和kubeconfig文件

rm -f /opt/kubernetes/cfg/kubelet.kubeconfig 
rm -f /opt/kubernetes/ssl/kubelet*

#这些文件是审批后自动生成的，所以每个节点的文件都是不一样的

更改kubelet的配置文件

vi /opt/kubernetes/cfg/kubelet.conf

--hostname-override=k8s-node01   #所有work节点都更改为自身对应的主机名称

更改kube-proxy的定义参数文件 

vi /opt/kubernetes/cfg/kube-proxy-config.yml

hostnameOverride: k8s-node01   #所有work节点都更改为自身对应的主机名称

所有work节点启动kubelet和kube-proxy服务

systemctl daemon-reload
systemctl start kubelet kube-proxy
systemctl enable kubelet kube-proxy

 在master节点审批work节点请求，使得work节点加入集群

kubectl get csr  #查看审批请求，加审批请求的名称填入以下审批命令中
kubectl certificate approve node-csr-RWx2DiMmRgLi8IsrpzwHthGve2W_YpIk3cccs8ZLNIk  #审批请求
kubectl get node  查看集群会发现，节点已加入集群

部署coredns组件

参考：k8s部署coredns_k8s pod怎么绑定coredns-CSDN博客 

至此k8s集群部署完成

集群网络功能验证

 部署busybox

vi busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

创建pod

kubectl apply -f busybox.yaml

进入pod验证

kubectl exec -it bosybox /bin/sh
nslookup kubernetes

nslookup kube-dns.kube-system

集群网络解析正常 文章来源地址https://www.toymoban.com/news/detail-839309.html

到了这里，关于k8s1.23.15集群二进制部署的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！