Keepalived-LVS 能够提高集群的高可用性并增加后端检测功能、简化配置,满足常规需求。但Keepalived-LVS集群中,同一个VIP只能由一台设备进行宣告,为一主多备的架构,不能横向拓展集群的性能,为此我们引入OSPF来解决该问题。
OSPF(ECMP)
ECMP(Equal-CostMultipathRouting)等价多路径,存在多条不同链路到达同一目的地址的网络环境中,如果使用传统的路由技术,发往该目的地址的数据包只能利用其中的一条链路,其它链路处于备份状态或无效状态,并且在动态路由环境下相互的切换需要一定时间,而等值多路径路由协议可以在该网络环境下同时使用多条链路,不仅增加了传输带宽,并且可以无时延无丢包地备份失效链路的数据传输。
特点:
- 基于流的均衡负载
- 最大链路数受设备限制(最高16)
- 所有链路都active,故障链路自动剔除
LVS+OSPF(ECMP)
利用ECMP以上特性,可以将LVS集群进行横向拓展,利用quagga启ospf
为模拟集群环境,我们准备了六台虚拟机分别为Client、LVS-1、LVS-2、RealServer1、RealServer2、Router,VIP设为192.168.0.100
- Router :192.168.0.1 192.168.1.1
- Client :192.168.1.2
- LVS-1 :192.168.0.2
- LVS-2 :192.168.0.3
- RealServer1 :192.168.0.4
- RealServer2 :192.168.0.5
Router 配置
LVS-1、LVS-2与Router需处于同一ospf域中,通过 IP 192.168.0.1 与LVS1/LVS2/RealServer1/RealServer2 通讯,IP 192.168.1.1 与 Client通讯
[root@router ~]# echo 1 > /proc/sys/net/ipv4/ip_forward // 开启转发 [root@router ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::20c:29ff:fe6e:d10e prefixlen 64 scopeid 0x20<link> ether 00:0c:29:6e:d1:0e txqueuelen 1000 (Ethernet) RX packets 63921 bytes 5978914 (5.7 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 24354 bytes 2334494 (2.2 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens38: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::20c:29ff:fe6e:d122 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:6e:d1:22 txqueuelen 1000 (Ethernet) RX packets 60501 bytes 5206254 (4.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 18358 bytes 1432690 (1.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 1119 bytes 88568 (86.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1119 bytes 88568 (86.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 // ens33:192.168.0.1 与LVS1/LVS2/RealServer1/RealServer同一子网 , ens38:192.168.1.1 与Client同一子网 [root@LVS1 ~]# yum install quagga.x86_64 -y [root@router ~]# vi /etc/quagga/zebra.conf hostname Router password test enable password test log file /var/log/quagga/zebra.log service password-encryption interface ens33 interface ens38 access-list 1 permit 127.0.0.1 ip prefix-list ANY seq 5 permit 0.0.0.0/0 le 32 route-map ANY deny 10 match ip address prefix-list ANY ip protocol ospf route-map ANY line vty access-class 1 [root@router ~]# vi /etc/quagga/ospfd.conf hostname Router password test log file /var/log/quagga/ospfd.log log stdout log syslog service password-encryption interface ens33 ip ospf hello-interval 1 ip ospf dead-interval 4 ip ospf priority 1 ip ospf cost 1 router ospf ospf router-id 192.168.0.7 log-adjacency-changes network 192.168.0.0/24 area 0.0.0.0 network 192.168.1.0/24 area 0.0.0.0 access-list 1 permit 127.0.0.1 line vty access-class 1
RealServer 配置
realserver.sh 参考上一章节
[root@RealServer1 ~]# ./realserver.sh start [root@RealServer1 ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::20c:29ff:febd:38da prefixlen 64 scopeid 0x20<link> ether 00:0c:29:bd:38:da txqueuelen 1000 (Ethernet) RX packets 48635 bytes 4087456 (3.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 46101 bytes 5700308 (5.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 30 bytes 2613 (2.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 30 bytes 2613 (2.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.0.100 netmask 255.255.255.255 loop txqueuelen 1000 (Local Loopback)
LVS 配置
由于我们使用ospf来实现高可用,不开启keepalived的vrrp功能(LVS2也同样配置),只是用其后端检测功能。
在keepalived-LVS集群的抢占模式下,Master在网卡上挂VIP并进行ARP广播,此时VIP对应的设备是唯一的。
但在OSPF-LVS集群中,Router根据ospf信息通过修改报文的目的mac地址转发到对应的LVS来实现均衡负载,并不根据VIP对应的ARP信息,所以对应的每台LVS将VIP挂在在lo上。
[root@LVS1 ~]# vi /etc/keepalived/keepalived.conf global_defs { router_id LVS1 #路由器标识 script_user root enable_script_security } virtual_server 192.168.0.100 80 { delay_loop 5 lb_algo wrr lb_kind DR persistence_timeout 60 persistence_granularity 255.255.255.255 protocol tcp inhibit_on_failure on ha_suspend sorry_server 127.0.0.1 80 real_server 192.168.0.4 80 { weight 10 HTTP_GET{ url{ path / status_code 200 } connect_port 80 connect_timeout 2 retry 1 delay_before_retry 1 } } real_server 192.168.0.5 80 { weight 10 HTTP_GET{ url{ path / status_code 200 } connect_port 80 connect_timeout 2 retry 1 delay_before_retry 1 } } } [root@LVS1 ~]# systemctl reload keepalived [root@LVS1 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.100:80 wrr persistent 60 -> 192.168.0.4:80 Route 10 0 0 -> 192.168.0.5:80 Route 10 0 0 [root@LVS1 ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255 up //添加VIP到lo [root@LVS1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 192.168.0.100/32 scope global lo:0 valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:af:6b:f7 brd ff:ff:ff:ff:ff:ff inet 192.168.0.2/24 brd 192.168.0.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feaf:6bf7/64 scope link valid_lft forever preferred_lft forever [root@LVS1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 ens33 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
安装quagga并进行配置
[root@LVS1 ~]# yum install quagga.x86_64 -y [root@LVS1 ~]# vi /etc/quagga/zebra.conf hostname LVS1 # HOSTNAME改为IP也可以 password test enable password test log file /var/log/quagga/zebra.log #log syslog service password-encryption interface ens33 access-list 1 permit 127.0.0.1 ip prefix-list ANY seq 5 permit 0.0.0.0/0 le 32 route-map ANY deny 10 match ip address prefix-list ANY ip protocol ospf route-map ANY line vty access-class 1 [root@LVS1 ~]# vi /etc/quagga/ospfd.conf hostname LVS1 password test log file /var/log/quagga/ospfd.log log stdout log syslog service password-encryption interface ens33 ip ospf hello-interval 1 ip ospf dead-interval 4 ip ospf priority 0 ip ospf cost 1 router ospf ospf router-id 192.168.0.2 log-adjacency-changes network 192.168.0.2/24 area 0.0.0.0 access-list 1 permit 127.0.0.1 line vty access-class 1 [root@LVS1 ~]# systemctl start zebra [root@LVS1 ~]# systemctl start ospfd
同理我们配置完LVS2后进行检测ospf状态
[root@LVS1 ~]# vtysh Hello, this is Quagga (version 0.99.22.4). Copyright 1996-2005 Kunihiro Ishiguro, et al. LVS1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL 192.168.0.7 1 Full/DR 3.817s 192.168.0.1 ens33:192.168.0.2 0 0 0 192.168.0.3 0 2-Way/DROther 3.518s 192.168.0.3 ens33:192.168.0.2 0 0 0 LVS1# show ip ospf route ============ OSPF network routing table ============ N 192.168.0.0/24 [1] area: 0.0.0.0 directly attached to ens33 N 192.168.0.100/32 [1] area: 0.0.0.0 directly attached to lo N 192.168.1.0/24 [2] area: 0.0.0.0 via 192.168.0.1, ens33 ============ OSPF router routing table ============= ============ OSPF external routing table ===========
在Route上查看路由文章来源:https://www.toymoban.com/news/detail-671056.html
router# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, A - Babel, > - selected route, * - FIB route K>* 0.0.0.0/0 via 192.168.0.10, ens33 C>* 127.0.0.0/8 is directly connected, lo O 192.168.0.0/24 [110/1] is directly connected, ens33, 00:11:04 C>* 192.168.0.0/24 is directly connected, ens33 O 192.168.0.100/32 [110/2] via 192.168.0.2, ens33 inactive, 00:00:11 via 192.168.0.3, ens33 inactive, 00:00:11 O 192.168.1.0/24 [110/1] is directly connected, ens38, 00:12:19 C>* 192.168.1.0/24 is directly connected, ens38 //可以看到 到 192.168.0.100/32 的下一跳分配到 192.168.0.2、192.168.0.3
Client 测试
[root@Client ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 ens33 192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 [root@Client ~]# traceroute 192.168.0.100 traceroute to 192.168.0.100 (192.168.0.100), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 0.575 ms 0.258 ms 0.478 ms 2 192.168.0.100 (192.168.0.100) 1.901 ms 1.746 ms 1.370 ms [root@Client ~]# curl 192.168.0.100 RealServer1 192.168.0.4
自动化
以上是最基础的ospf-Lvs集群配置,在实际生产环境中会有自动化部署、监控、告警等需求,例如quagga的初始化配置文章来源地址https://www.toymoban.com/news/detail-671056.html
quagga 配置
HOSTNAME=`hostname` PASSWORD=shenyangchangkuan #获取默认路由对应网卡 NIC=`/sbin/route -n|awk '$1=="default"||$1=="0.0.0.0"{print $NF}'|head -n 1` if [ -z $NIC ];then echo "get NIC err,NIC is null!" fi #获取该网卡IP #centos7改为下面这行 #IP=`/sbin/ifconfig $NIC |grep "inet "|sed 's/:/ /'|awk '{print $2}'` IP=`/sbin/ifconfig $NIC|grep "inet addr"|sed 's/:/ /'|awk '{print $3}'` if [ -z $IP ];then echo "get IP err,IP is null!" exit 1 fi #将该网卡及子网卡IP信息写入tmp/lvs_network.tmp echo "">/tmp/lvs_network.tmp for cfg in ` ls -l /etc/sysconfig/network-scripts/ifcfg-${NIC}*|awk '{print $NF}'` do cat $cfg |grep IPADDR=|awk -F "=" '{print $2}' |sed 's/"//g'>>/tmp/lvs_network.tmp done
到了这里,关于Ceph入门到精通-大流量10GB/s支持OSPF(ECMP)-LVS 集群的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!