1 部署Nginx+Keepalived高可用负载均衡器
可参考以下链接,了解nginx和keepalived,不看也不影响部署
CHAPTER 3 Web HA集群部署 - Keepalived
CHAPTER 1 Web Server - nginx 安装配置
- Nginx是一个主流Web服务和反向代理服务器,这里用四层实现对apiserver实现负载均衡。
- Keepalived是一个主流高可用软件,基于VIP绑定实现服务器双机热备,在上述拓扑中,Keepalived主要根据Nginx运行状态判断是否需要故障转移(漂移VIP),例如当Nginx主节点挂掉,VIP会自动绑定在Nginx备节点,从而保证VIP一直可用,实现Nginx高可用。
- 如果你是在公有云上,一般都不支持keepalived,那么你可以直接用它们的负载均衡器产品,直接负载均衡多台Master kube-apiserver,架构与上面一样。
在两台Master节点操作
1.1 安装软件包(Master1/Master2)
yum install epel-release -y
yum install nginx keepalived -y
1.2 Nginx配置文件(主备相同)
cat > /etc/nginx/nginx.conf << "EOF"
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡,为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.71.251:6443; # Master1 APISERVER IP:PORT
server 192.168.71.254:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用,这个监听端口不能是6443,否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
EOF
1.3 keepalived配置文件(Master1)
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 100 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.71.250/24
}
track_script {
check_nginx
}
}
EOF
- vrrp_script:指定检查nginx工作状态脚本(根据nginx状态判断是否故障转移)
- virtual_ipaddress:虚拟IP(VIP)
准备上述配置文件中检查Nginx运行状态的脚本
cat > /etc/keepalived/check_nginx.sh << "EOF"
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
1.4 keepalived配置(master2)
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.71.250/24
}
track_script {
check_nginx
}
}
EOF
准备上述配置文件中检查nginx运行状态的脚本:
cat > /etc/keepalived/check_nginx.sh << "EOF"
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
说明:
keepalived根据脚本返回状态码(0为工作正常,非0不正常)判断是否故障转移。
1.5 Nginx增加Steam模块
1.5.1 查看Nginx版本模块
如果已经安装 --with-stream模块,后面的步骤可以跳过
[root@k8s-master2 nginx-1.20.1]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --with-stream
1.5.2 下载同一个版本的nginx
下载地址 : http://nginx.org/download/
1.5.3 备份原Nginx文件
mv /usr/sbin/nginx /usr/sbin/nginx.bak
cp -r /etc/nginx{,.bak}
1.5.4 重新编译Nginx
检查模块是否支持,比如这次添加 limit 限流模块 和 stream 模块:
./configure –help | grep limit
ps:-without-http_limit_conn_module disable 表示已有该模块,编译时,不需要添加
./configure –help | grep stream
ps:–with-stream enable 表示不支持,编译时要自己添加该模块
根据第1步查到已有的模块,加上本次需新增的模块: --with-stream
编译环境准备
yum -y install libxml2 libxml2-dev libxslt-devel
yum -y install gd-devel
yum -y install perl-devel perl-ExtUtils-Embed
yum -y install GeoIP GeoIP-devel GeoIP-data
yum -y install pcre-devel
yum -y install openssl openssl-devel
yum -y install gcc make
编译
tar -xf nginx-1.20.1.tar.gz
cd nginx-1.20.1/
./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --with-stream
make
说明:
make完成后不要继续输入“make install”,以免现在的nginx出现问题
以上完成后,会在objs目录下生成一个nginx文件,先验证:
[root@k8s-master2 nginx-1.20.1]# ./objs/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
1.5.5 替换nginx到Master1/Master2
cp ./objs/nginx /usr/sbin/
scp objs/nginx root@192.168.242.51:/usr/sbin/
1.5.6 修改nginx服务文件
vim /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/bin/rm -rf /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecStop=/usr/sbin/nginx -s stop
ExecReload=/usr/sbin/nginx -s reload
PrivateTmp=true
[Install]
WantedBy=multi-user.target
1.6 启动并设置开机自启(master1/master2)
systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived
nginx无法启动
[root@node-254 ~]# /usr/sbin/nginx -t
nginx: [emerg] unknown directive "stream" in /etc/nginx/nginx.conf:13
nginx: configuration file /etc/nginx/nginx.conf test failed
解决办法:
# 安装nginx源
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
# 先安装
yum -y install epel-release
#应该是缺少modules模块
yum -y install nginx-all-modules.noarch
然后在用nginx -t就好了
[root@k8s-node2 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
1.7 查看keepalived工作状态
[root@node-251 kubernetes]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:84:07:56 brd ff:ff:ff:ff:ff:ff
inet 192.168.71.251/24 brd 192.168.71.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.71.250/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::ca6d:22ce:c36:f87b/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::7fd7:bd03:f1c1:ee7c/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::a2f8:7cd5:7ddb:3558/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:08:fa:41:ae brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.16.52.192/32 scope global tunl0
valid_lft forever preferred_lft forever
5: calid70e12abeda@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
可以看到,在ens33网卡绑定了192.168.242.55 虚拟IP,说明工作正常。
1.8 Nginx+keepalived高可用测试
关闭主节点Nginx,测试VIP是否漂移到备节点服务器。
在Nginx Master执行 pkill nginx;
在Nginx Backup,ip addr命令查看已成功绑定VIP。
[root@node-254 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:c6:83:89 brd ff:ff:ff:ff:ff:ff
inet 192.168.70.251/24 brd 192.168.70.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.71.250/24 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fec6:8389/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:b6:fb:9c:83 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: docker_gwbridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:dd:b0:70:37 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
valid_lft forever preferred_lft forever
5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.16.212.128/32 scope global tunl0
valid_lft forever preferred_lft forever
1.9 访问负载均衡器测试
找K8s集群中任意一个节点,使用curl查看K8s版本测试,使用VIP访问:
[root@node-251 kubernetes]# curl -k https://192.168.71.250:16443/version
{
"major": "1",
"minor": "20",
"gitVersion": "v1.20.15",
"gitCommit": "8f1e5bf0b9729a899b8df86249b56e2c74aebc55",
"gitTreeState": "clean",
"buildDate": "2022-01-19T17:23:01Z",
"goVersion": "go1.15.15",
"compiler": "gc",
"platform": "linux/amd64"
可以正确获取到K8s版本信息,说明负载均衡器搭建正常。该请求数据流程:curl -> vip(nginx) -> apiserver
通过查看Nginx日志也可以看到转发apiserver IP:
[root@node-251 kubernetes]# tailf /var/log/nginx/k8s-access.log
192.168.71.251 192.168.71.251:6443 - [09/May/2023:05:18:17 -0400] 200 424
192.168.20.252 192.168.71.251:6443 - [09/May/2023:05:19:24 -0400] 200 541
192.168.71.251 192.168.71.251:6443 - [09/May/2023:05:22:07 -0400] 200 424
正常可以看到访问的IP地址在71.251和70.251切换,但是笔者的两个master不在同一网段,所以无法切换
1.10 修改所有的Work Node连接LB VIP
试想下,虽然我们增加了Master2 Node和负载均衡器,但是我们是从单Master架构扩容的,也就是说目前所有的Worker Node组件连接都还是Master1 Node,如果不改为连接VIP走负载均衡器,那么Master还是单点故障。
因此接下来就是要改所有Worker Node(kubectl get node命令查看到的节点)组件配置文件
由原来192.168.71.251修改为192.168.71.250(VIP)。
在所有Worker Node执行:
sed -i 's#192.168.71.251:6443#192.168.71.250:16443#' /opt/kubernetes/cfg/*
systemctl restart kubelet kube-proxy
检查节点状态
[root@node-251 kubernetes]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-251 Ready <none> 124m v1.20.15
node-252 Ready <none> 84m v1.20.15
node-253 Ready <none> 84m v1.20.15
node-254 Ready <none> 65m v1.20.15
至此,一套高可用的k8s二进制可用集群就部署完成了~文章来源:https://www.toymoban.com/news/detail-441758.html
如果出现 x509: certificate is valid for 10.0.0.1, 127.0.0.1, 192.168.71.251, 192.168.71.252, 192.168.71.253, 192.168.70.251, not 192.168.71.250
则可能是生成证书的时候,没有提前设置虚拟IP,所以无法高可用,这种情况需要重新生成证书,或者再开始的时候策划好服务器IP等相关内容文章来源地址https://www.toymoban.com/news/detail-441758.html
到了这里,关于(六)Kubernetes - 手动部署(二进制方式安装)的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!