基于本地存储LVM新建虚机方案
date: 2021/12/22
auth: mmwei3
一、环境信息如下：
| IP | 角色 | VGS |
|---|---|---|
| 192.168.162.123 | 控制+计算+存储 | cinder-volumes、cinder-volumes-ssd |
| 192.168.162.124 | 控制+计算+存储 | cinder-volumes、cinder-volumes-ssd |
| 192.168.162.125 | 控制+计算+存储 | cinder-volumes、cinder-volumes-ssd |
二、需求方案：
1、虚机(卷启动)+系统盘+数据盘 三者在同一计算节点。
2、虚机可以挂本计算节点的数据盘也可以挂载其他计算节点的数据盘。
3、虚机可以使用本节点上的HDD做系统盘及云盘，也可以使用本节点上的SSD做系统盘及云盘
4、虚机(本地)+数据盘 三者在同一计算节点。
5、虚机冷迁移方案
三、环境部署部分说明
参考官方文档：
https://docs.openstack.org/mitaka/config-reference/block-storage/drivers/lvm-volume-driver.html
https://docs.openstack.org/cinder/latest/admin/blockstorage-lio-iscsi-support.html
1、每个节点新建两个vg：cinder-volumes、cinder-volumes-ssd
# 001节点
[root@rg2-test-control001 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
cinder-volumes 1 4 0 wz--n- <3.64t <3.30t
cinder-volumes-ssd 2 0 0 wz--n- 278.24g 278.24g
# 002节点
[root@rg2-test-control002 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
cinder-volumes 1 1 0 wz--n- <3.64t <3.59t
cinder-volumes-ssd 2 0 0 wz--n- 278.24g 278.24g
# 003节点
[root@rg2-test-control003 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
cinder-volumes 1 1 0 wz--n- <3.64t <3.59t
cinder-volumes-ssd 2 2 0 wz--n- 278.24g 258.24g
2、修改相关配置文件 (解决方案一)
参考官方文档：
https://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html
https://docs.openstack.org/mitaka/config-reference/compute/iscsi-offload.html
https://docs.openstack.org/cinder/latest/admin/ts-cinder-config.html
https://docs.openstack.org/cinder/latest/admin/blockstorage-multi-backend.html
① 修改cinder-volume、cinder-api、cinder-scheduler配置
# 三个节点均需修改(或者说cinder-volume节点所在节点都修改)
# 为了保持统一cinder-api、cinder-scheduler、cinder-volume三者均修改
# iSCSI target user-land tool to use. tgtadm is default, use
# lioadm for LIO iSCSI support, scstadmin for SCST target
# support, iseradm for the ISER protocol, ietadm for iSCSI
# Enterprise Target, iscsictl for Chelsio iSCSI Target or fake
# for testing. (string value)
#iscsi_helper=tgtadm
# Method used to wipe old volumes (string value)
#volume_clear=zero
# Size in MiB to wipe at start of old volumes. 0 => all
# (integer value)
#volume_clear_size=0
[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf|egrep -w 'enabled_backends|default_volume_type'
enabled_backends = lvm,lvm-ssd
default_volume_type = lvm
[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf | grep 'storage_availability_zone'
storage_availability_zone = rg2-test-control003.ostack.hfb3.iflytek.net (cinder-volume节点的主机名, 即myhostname)
[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf | egrep -w '\[lvm\]' -A 30
[lvm]
volume_backend_name = lvm
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes1
iscsi_protocol = iscsi
#iscsi_helper = lioadm #这里不适用lioadm，使用默认的配置 #iscsi_helper=tgtadm,可以不写
volume_clear = zero
[lvm-ssd]
volume_backend_name = lvm-ssd
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes-ssd
iscsi_protocol = iscsi
#iscsi_helper = lioadm
volume_clear = zero
[root@rg2-test-control003 ~]#
② 修改glance-api配置
参考官方文档：https://docs.openstack.org/glance/pike/install/install-rdo.html
# 三个节点均需修改(或者说glance-api节点所在节点都修改)
[root@rg2-test-control003 ~]# cat /etc/kolla/glance-api/glance-api.conf| grep '\[glance_store\]' -A 10
[glance_store]
#default_store = rbd
#filesystem_store_datadir = /var/lib/glance/images/
#stores = rbd
#rbd_store_user = glance
#rbd_store_pool = images
#rbd_store_ceph_conf = /etc/ceph/ceph.conf
#rbd_volume_pool = sata_pool
default_store = file
filesystem_store_datadir = /var/lib/glance/images/
stores = file
② 修改nova-compute配置
[root@rg2-test-control003 ~]# cat /etc/kolla/nova-compute/nova.conf | grep '\[libvirt\]' -A 10
[libvirt]
connection_uri = qemu+tcp://192.168.162.125/system
#images_rbd_pool = vms
#images_type = rbd
#images_rbd_ceph_conf = /etc/ceph/ceph.conf
#rbd_user = nova
#rbd_secret_uuid = df5fbfad-02c7-43b2-9938-900c80d6bc99
images_type = lvm
images_volume_group=cinder-volumes1
3、修改相关配置文件 (解决方案二)
参考官方文档：
https://docs.openstack.org/cinder/pike/scheduler-filters.html
https://github.com/openstack/cinder/blob/master/cinder/scheduler/filters/instance_locality_filter.py
https://opendev.org/openstack/cinder/src/commit/40a6fff6d408f2ef0317c08fb165cc75a92c380a/setup.cfg
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/
openstack_data_processing/instancelocality
https://specs.openstack.org/openstack/sahara-specs/specs/kilo/volume-instance-locality.html
# 在方案一的基础上新增如下配置
# 该配置可以尝试使我们的虚机实例和系统盘在同一个节点上创建
[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-scheduler/cinder.conf | grep scheduler_default_filters -A 3
scheduler_default_filters=AvailabilityZoneFilter,CapacityFilter,CapabilitiesFilter,InstanceLocalityFilter
volume_local_to_instance = True # value for true or false
四、运行相关容器
# iscsi-initiator-utils scsi-target-utils.x86_64
# cinder-volume容器挂载一下-v iscsi_info:/etc/iscsi nova-compute也挂在一下
cat /etc/kolla/iscsid/config.json
{
"command": "iscsid -d 8 -f --pid=/run/iscsid.pid",
"config_files": []
}
cat /etc/kolla/tgtd/config.json
{
"command": "tgtd -d 1 -f --iscsi portal=192.168.162.123:3260",
"config_files": []
}
# nova_compute
docker run \
-d --tty=true --net=host --restart=always \
--privileged \
--name=nova_compute \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=nova-compute \
-v /etc/kolla/nova-compute/:/var/lib/kolla/config_files/:ro \
-v /etc/localtime:/etc/localtime:ro \
-v /lib/modules:/lib/modules:ro \
-v /run:/run:shared \
-v /dev:/dev \
-v kolla_logs:/var/log/kolla/ \
-v libvirtd:/var/lib/libvirt \
-v nova_compute:/var/lib/nova/ \
-v iscsi_info:/etc/iscsi \
mmwei.com/lokolla/centos-source-nova-compute:15.0.6-ccr-1.1.3-priv.7-2
#cinder_volume
docker run \
-d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=cinder_volume \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /dev/:/dev/ \
-v /lib/modules:/lib/modules:ro \
-v /run/:/run/:shared \
-v cinder:/var/lib/cinder \
-v iscsi_info:/etc/iscsi \
-v /etc/kolla/cinder-volume/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-volume:10.0.3-ccr-1.0.6
docker run \
-d --tty=true --net=host --restart=always \
--name=cinder_api \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /etc/kolla/cinder-api/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-api:10.0.3-ccr-1.0.6
docker run \
-d --tty=true --net=host --restart=always \
--name=cinder_scheduler \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /etc/kolla/cinder-scheduler/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-scheduler:10.0.3-ccr-1.0.6
mmwei.com/lokolla/centos-source-cinder-volume:10.0.3-ccr-1.0.8
# iscsi-tgtd
docker run -d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=tgtd \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=tgtd \
-v /etc/kolla/tgtd/:/var/lib/kolla/config_files/:ro \
-v kolla_logs:/var/log/kolla/
-v /etc/localtime:/etc/localtime:ro \
-v /dev:/dev \
-v /run:/run:shared \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /lib/modules:/lib/modules:ro \
-v /sys/kernel/config:/configfs \
-v iscsi_info:/etc/iscsi \
mmwei.com/lokolla/centos-source-tgtd:train-1
# iscsid
docker run -d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=iscsid \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=iscsid \
-v /etc/kolla/iscsid/:/var/lib/kolla/config_files/:ro \
-v kolla_logs:/var/log/kolla/ \
-v /etc/localtime:/etc/localtime:ro \
-v /dev:/dev \
-v /run:/run:shared \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /lib/modules:/lib/modules:ro \
-v /sys/kernel/config:/configfs \
-v iscsi_info:/etc/iscsi \
mmwei.com/lokolla/centos-source-iscsid:train-v1
五、验证方案
1、虚机(卷启动)+系统盘+数据盘 三者在同一计算节点。
① 上传镜像
[root@rg2-test-control001 ~]# glance image-create
--disk-format raw \
--container-format bare \
--file /root/mmwei3/Centos-7.4.raw \
--property hw_qemu_guest_agent=yes \
--name Centos-7.4-lvm \
--visibility public \
--progress
② 新建主机聚合
# 先查看服务状态 是否正常，zone是否生效
[root@rg2-test-control001 ~]# openstack volume service list -c Binary -c Host -c Zone
+-----------------+----------------------------------------------------+---------------------------------------------+
| Binary | Host | Zone |
+-----------------+----------------------------------------------------+---------------------------------------------+
| cinder-scheduler| rg2-test-control002.ostack.hfb3.iflytek.net | nova |
| cinder-scheduler| rg2-test-control003.ostack.hfb3.iflytek.net | nova |
| cinder-scheduler| rg2-test-control001.ostack.hfb3.iflytek.net | nova |
| cinder-volume | rg2-test-control001.ostack.hfb3.iflytek.net@lvm | nova |
| cinder-volume | rg2-test-control003.ostack.hfb3.iflytek.net@lvm | rg2-test-control003.ostack.hfb3.iflytek.net |
| cinder-volume | rg2-test-control002.ostack.hfb3.iflytek.net@lvm | rg2-test-control002.ostack.hfb3.iflytek.net |
| cinder-volume | rbd:sata@rbd-sata | nova |
| cinder-volume | rg2-test-control003.ostack.hfb3.iflytek.net@lvm-ssd| rg2-test-control003.ostack.hfb3.iflytek.net |
+-----------------+----------------------------------------------------+---------------------------------------------+
[root@rg2-test-control001 ~]# nova aggregate-create
usage: nova aggregate-create <name> [<availability-zone>]
[root@rg2-test-control001 ~]# nova aggregate-create normal
[root@rg2-test-control001 ~]# nova aggregate-create lvm
[root@rg2-test-control001 ~]# nova aggregate-create lvm2
# 加入聚合
[root@rg2-test-control001 ~]# nova aggregate-add-host
usage: nova aggregate-add-host <aggregate> <host>
[root@rg2-test-control001 ~]# nova aggregate-add-host normal rg2-test-control001.ostack.hfb3.iflytek.net
[root@rg2-test-control001 ~]# nova aggregate-add-host lvm rg2-test-control002.ostack.hfb3.iflytek.net
[root@rg2-test-control001 ~]# nova aggregate-add-host lvm2 rg2-test-control003.ostack.hfb3.iflytek.net
[root@rg2-test-control001 ~]# nova aggregate-list
+----+--------+---------------------------------------------+--------------------------------------+
| Id | Name | Availability Zone | UUID |
+----+--------+---------------------------------------------+--------------------------------------+
| 1 | normal | nova | e1b4ddac-957f-49a8-ac08-0054a33edc3d |
| 9 | lvm | rg2-test-control002.ostack.hfb3.iflytek.net | 1beb7c26-a49e-4bdc-8469-c844c8dac756 |
| 11 | lvm2 | rg2-test-control003.ostack.hfb3.iflytek.net | 1ba48e46-6443-4557-9098-1bff0e0edb0f |
+----+--------+---------------------------------------------+--------------------------------------+
③ 新建卷类型
[root@rg2-test-control001 ~]# openstack volume type create --public --property volume_backend_name='lvm-ssd' lvm-ssd
[root@rg2-test-control001 ~]# openstack volume type create --public --property volume_backend_name='lvm' lvm
[root@rg2-test-control001 ~]# openstack volume type list
+--------------------------------------+---------+-----------+
| ID | Name | Is Public |
+--------------------------------------+---------+-----------+
| a2a42521-ab9c-42aa-b15a-2af147ce03d2 | lvm-ssd | True |
| 96a28f6a-c3b6-43fc-bd99-887daa579663 | lvm | True |
+--------------------------------------+---------+-----------+
[root@rg2-test-control001 ~]# openstack volume type show a2a42521-ab9c-42aa-b15a-2af147ce03d2
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| access_project_ids | None |
| description | None |
| id | a2a42521-ab9c-42aa-b15a-2af147ce03d2 |
| is_public | True |
| name | lvm-ssd |
| properties | volume_backend_name='lvm-ssd' |
| qos_specs_id | None |
+--------------------+--------------------------------------+
[root@rg2-test-control001 ~]# openstack volume type show 96a28f6a-c3b6-43fc-bd99-887daa579663
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| access_project_ids | None |
| description | None |
| id | 96a28f6a-c3b6-43fc-bd99-887daa579663 |
| is_public | True |
| name | lvm |
| properties | volume_backend_name='lvm' |
| qos_specs_id | None |
+--------------------+--------------------------------------+
④ 新建卷及云盘
# 1、从镜像创建卷(作为系统盘)，指定availability-zone
[root@rg2-test-control001 ~]# openstack volume create \
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100 \
--type lvm --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
vol-zone2-os1
+---------------------+---------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------+
| attachments | [] |
| availability_zone | rg2-test-control002.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-12-23T03:49:10.027995 |
| description | None |
| encrypted | False |
| id | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b |
| migration_status | None |
| multiattach | False |
| name | vol-zone2-os1 |
| properties | |
| replication_status | None |
| size | 100 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | lvm |
| updated_at | None |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
+---------------------+---------------------------------------------+
### 时间问题 耗时约1分钟
相关日志
[root@rg2-test-control00 ~]# tail -f /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log
2021-12-23 10:25:29.012 31 INFO cinder.volume.flows.manager.create_volume\
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \
dd94c9e5d29048b68b56131a0d6ea2ba - default default] \
Volume 1661215d-1302-4065-8e80-7f3e29591742: being created as image with specification:\
{'status': u'creating', 'image_location': (u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690',\
[{u'url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690', u'metadata': {}}]),\
'volume_size': 100, 'volume_name': 'volume-1661215d-1302-4065-8e80-7f3e29591742', \
'image_id': '2e3c5301-e2ed-4357-884e-f28ad0428690', 'image_service': \
<cinder.image.glance.GlanceImageService object at 0x7fb5f0d09a50>, \
'image_meta': {u'status': u'active', u'file': u'/v2/images/2e3c5301-e2ed-4357-884e-f28ad0428690/file',\
u'virtual_size': None, u'name': u'Centos-7.4-lvm', u'tags': [], \
u'container_format': u'bare', u'created_at': datetime.datetime(2021, 12, 20, 14, 21, 13, tzinfo=<iso8601.Utc>),\
u'disk_format': u'raw',\
u'locations': [{u'url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690', u'metadata': {}}], \
u'visibility': u'public', u'updated_at': datetime.datetime(2021, 12, 20, 14, 21, 37, tzinfo=<iso8601.Utc>), \
u'owner': u'dd94c9e5d29048b68b56131a0d6ea2ba', u'protected': False, u'id': u'2e3c5301-e2ed-4357-884e-f28ad0428690', \
u'min_ram': 0, u'checksum': u'afcb6cdfb41dd41343c122915c741026',\
u'min_disk': 0, u'direct_url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690',\
'properties': {u'hw_qemu_guest_agent': u'yes'}, u'size': 5368709120}}
2021-12-23 10:25:52.487 31 INFO cinder.image.image_utils \
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad\
dd94c9e5d29048b68b56131a0d6ea2ba - default default] Image download 5120.00 MB at 218.14 MB/s
2021-12-23 10:26:31.357 31 INFO cinder.image.image_utils\
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \
dd94c9e5d29048b68b56131a0d6ea2ba - default default] Converted 5120.00 MB image at 172.83 MB/s \
2021-12-23 10:26:33.556 31 INFO cinder.volume.flows.manager.create_volume \
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \
dd94c9e5d29048b68b56131a0d6ea2ba - default default] \
Volume volume-1661215d-1302-4065-8e80-7f3e29591742 (1661215d-1302-4065-8e80-7f3e29591742): \
created successfully \
2021-12-23 10:26:33.561 31 INFO cinder.volume.manager \
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \
dd94c9e5d29048b68b56131a0d6ea2ba - default default] Created volume successfully.
# 2、 新建空白卷 耗时秒级 5s以内
[root@rg2-test-control001 ~]# openstack volume create \
--size 8 --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
test-volume-vdb
+---------------------+---------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------+
| attachments | [] |
| availability_zone | rg2-test-control002.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-12-23T03:50:53.717870 |
| description | None |
| encrypted | False |
| id | e56a2896-5f59-4789-8ff6-f272683ffbfa |
| migration_status | None |
| multiattach | False |
| name | test-volume-vdb |
| properties | |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | lvm |
| updated_at | None |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
+---------------------+---------------------------------------------+
# 3、 新建虚机
[root@rg2-test-control001 ~]# openstack flavor list
+--------------------------------------+------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+------------+------+------+-----------+-------+-----------+
| 4bea2583-f630-430a-95cf-4665414f8a78 | t4.4medium | 4096 | 50 | 0 | 4 | True |
+--------------------------------------+------------+------+------+-----------+-------+-----------+
[root@rg2-test-control001 ~]# openstack network list
+--------------------------------------+-----------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-----------------------+--------------------------------------+
| bd23d0fd-e513-4f4e-a901-ef01b0476f43 | Classical_Network_163 | 1a5aefa9-aa0d-40e7-af4e-78d34f146346 |
+--------------------------------------+-----------------------+--------------------------------------+
# 这种启动方式 可以保证虚机qemu进程及系统盘、数据盘三者都在同一台机器上
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--block-device source=volume,id=e56a2896-5f59-4789-8ff6-f272683ffbfa,dest=volume,shutdown=preserve \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 \
--availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
test-zone2-1
# 如果不需要数据盘则：
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 \
--availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
test-zone2-1
#######------------------------------如果采用方案二 加入 InstanceLocalityFilter 过滤器--------------
# 此方案适用于新建虚机不建数据盘或者数据盘在其他节点或者其他存储池
# 且volume_local_to_instance = True 则新建虚机时不需要指定zone,新建系统盘时 也不需要指定zone
# 但是新建数据盘需要指定zone
# 如果新建虚机不加数据盘则直接新建便可
[root@rg2-test-control001 ~]# openstack volume create \
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100 \
--type lvm vol-zone2-os1
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 \
test-zone2-1
# 如果新建虚机同时需要加入数据盘，则 新建系统盘时 需要指定zone,这样新建数据盘便可以更好的识别zone.
[root@rg2-test-control001 ~]# openstack volume create \
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100 \
--type lvm --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
vol-zone2-os1
[root@rg2-test-control001 ~]# openstack volume create \
--size 8 --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \
test-volume-vdb
2、虚机可以挂本计算节点的数据盘也可以挂载其他计算节点的数据盘。
[root@rg2-test-control001 ~]# nova volume-attachments 368c291b-981e-4fef-90ce-643397b00870
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID | DEVICE | SERVER ID | VOLUME ID |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 76a5ef99-2888-4fb9-99f0-8b3a23e4854b | /dev/vda | 368c291b-981e-4fef-90ce-643397b00870 | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b |
| e56a2896-5f59-4789-8ff6-f272683ffbfa | /dev/vdb | 368c291b-981e-4fef-90ce-643397b00870 | e56a2896-5f59-4789-8ff6-f272683ffbfa |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
[root@rg2-test-control001 ~]#
[root@rg2-test-control001 ~]# cinder show e56a2896-5f59-4789-8ff6-f272683ffbfa
+--------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------+
| attached_servers | ['368c291b-981e-4fef-90ce-643397b00870'] |
| attachment_ids | ['3bd98c3a-3501-428e-8044-293f028297b1'] |
| availability_zone | rg2-test-control002.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-12-23T03:50:54.000000 |
| description | None |
| encrypted | False |
| id | e56a2896-5f59-4789-8ff6-f272683ffbfa |
| metadata | attached_mode : rw |
| migration_status | None |
| multiattach | False |
| name | test-volume-vdb |
| os-vol-host-attr:host | rg2-test-control002.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | dd94c9e5d29048b68b56131a0d6ea2ba |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2021-12-23T03:53:48.000000 |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
| volume_type | lvm |
+--------------------------------+-----------------------------------------------------+
[root@rg2-test-control001 ~]# openstack volume create --size 8 \
--availability-zone rg2-test-control003.ostack.hfb3.iflytek.net \
test-volume-vdb
+---------------------+---------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------+
| attachments | [] |
| availability_zone | rg2-test-control003.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-12-23T04:08:01.380129 |
| description | None |
| encrypted | False |
| id | 8a6d7913-6253-4f44-b654-186ae87ed886 |
| migration_status | None |
| multiattach | False |
| name | test-volume-vdb |
| properties | |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | lvm |
| updated_at | None |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
+---------------------+---------------------------------------------+
[root@rg2-test-control001 ~]# nova volume-attach \
368c291b-981e-4fef-90ce-643397b00870 \
8a6d7913-6253-4f44-b654-186ae87ed886
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 8a6d7913-6253-4f44-b654-186ae87ed886 |
| serverId | 368c291b-981e-4fef-90ce-643397b00870 |
| volumeId | 8a6d7913-6253-4f44-b654-186ae87ed886 |
+----------+--------------------------------------+
[root@rg2-test-control001 ~]# nova volume-attachments 368c291b-981e-4fef-90ce-643397b00870
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID | DEVICE | SERVER ID | VOLUME ID |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 76a5ef99-2888-4fb9-99f0-8b3a23e4854b | /dev/vda | 368c291b-981e-4fef-90ce-643397b00870 | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b |
| e56a2896-5f59-4789-8ff6-f272683ffbfa | /dev/vdb | 368c291b-981e-4fef-90ce-643397b00870 | e56a2896-5f59-4789-8ff6-f272683ffbfa |
| 8a6d7913-6253-4f44-b654-186ae87ed886 | /dev/vdc | 368c291b-981e-4fef-90ce-643397b00870 | 8a6d7913-6253-4f44-b654-186ae87ed886 |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
3、虚机可以使用本节点上的HDD做系统盘及云盘，也可以使用本节点上的SSD做系统盘及云盘
# 仅需要将--type lvm 修改为 --type lvm-ssd便可
[root@rg2-test-control003 ~]# openstack volume create \
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100 \
--type lvm-ssd vol-zone2-vdb
# 如
[root@rg2-test-control003 ~]# cinder show 6e426442-967c-4c32-bb90-e82ffb6428c7
+--------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------------------+
| attached_servers | ['ec164258-db3f-43b2-a7cd-2e62a0c0af41'] |
| attachment_ids | ['59b8212f-87c5-45e3-b587-b76261f4fb25'] |
| availability_zone | rg2-test-control003.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-12-22T08:05:16.000000 |
| description | None |
| encrypted | False |
| id | 6e426442-967c-4c32-bb90-e82ffb6428c7 |
| metadata | attached_mode : rw |
| migration_status | None |
| multiattach | False |
| name | lvm-ssd-vdb |
| os-vol-host-attr:host | rg2-test-control003.ostack.hfb3.iflytek.net@lvm-ssd#lvm-ssd |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | dd94c9e5d29048b68b56131a0d6ea2ba |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2021-12-22T08:15:33.000000 |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
| volume_type | lvm-ssd |
+--------------------------------+-------------------------------------------------------------+
4、虚机(本地盘)+数据盘 三者在同一计算节点。(在方案1的基础上调整)
① 修改nova-compute配置
[root@rg2-test-control003 ~]# cat /etc/kolla/nova-compute/nova.conf | grep '\[libvirt\]' -A 10
[libvirt]
connection_uri = qemu+tcp://192.168.162.125/system
#images_rbd_pool = vms
#images_type = rbd
#images_rbd_ceph_conf = /etc/ceph/ceph.conf
#rbd_user = nova
#rbd_secret_uuid = df5fbfad-02c7-43b2-9938-900c80d6bc99
#images_type = lvm
#images_volume_group=cinder-volumes1
# 新建虚机
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78\
--block-device source=image,id=2e3c5301-e2ed-4357-884e-f28ad0428690,size=50,bootindex=0,dest=local \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 instance_lvm001
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78\
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 instance_lvm002
# 新建空白卷，根据新建的虚机分配的zone
[root@rg2-test-control001 ~]# openstack volume migrate \
--host rg2-test-control001.ostack.hfb3.iflytek.net d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
[root@rg2-test-control001 ~]# nova volume-attach \
f72d7b2e-12f4-4857-89d5-177bc845e15f d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
[root@rg2-test-control001 ~]# nova volume-attachments f72d7b2e-12f4-4857-89d5-177bc845e15f
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID | DEVICE | SERVER ID | VOLUME ID |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 | /dev/vdb | f72d7b2e-12f4-4857-89d5-177bc845e15f | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
5、虚机迁移方案
#冷迁移(cold migration)，也叫静态迁移。关闭电源的虚拟机进行迁移。通过冷迁移，\
#可以选择将关联的磁盘从一个数据存储移动到另一个数据存储。
#好处：虚拟机不需要位于共享存储器上，数据丢失率小。
#缺点：需要关闭电源，业务中断。
#虚拟机的迁移，就是数据的转移，如果计算节点之间没有共享存储，所以要转移的数据包括两部分：
#1、静态数据：存储在本地的虚拟机的镜像文件，包括后端镜像(libvirt Base)和虚拟机单独的增量镜像文件(libvirt instance disks)。
#2、动态数据：内存里虚拟机的运行时数据，内存里的数据是动态变化的数据，虚拟机里运行的负载的大小直接影响迁移的时间长短。
文章来源：https://www.toymoban.com/news/detail-712413.html
文章来源地址https://www.toymoban.com/news/detail-712413.html
① 共享存储与非共享存储
# 虚拟机的数据存在共享磁盘上(Shared storage-based live migration)，迁移只需要完成内存数据的迁移。
# 虚拟机的数据存在本地磁盘(block migration)，需要对镜像文件和内存数据同时迁移。
② 冷迁移
# 冷迁移实现方法有多种，例如有快照来迁移实例、实例文件迁移。以文件迁移为例，完成冷迁移。
# 虚机冷迁移步骤：
#1、关闭虚拟机
#2、找到虚拟机位于/var/lib/nova/instances下文件
#3、将虚拟机的文件全部copy到目标主机的相同位置下
#4、修改用户组
#5、更新数据库中host,node字段为目标主机的名字
#6、重启目标主机的nova-compute服务
# 测试虚机
[root@rg2-test-control001 ~]# nova live-migration --block-migrate f72d7b2e-12f4-4857-89d5-177bc845e15f
# 迁移卷
[root@rg2-test-control001 ~]# cinder migrate \
d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 rg2-test-control001.ostack.hfb3.iflytek.net@lvm#lvm
[root@rg2-test-control001 ~]# cinder show d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
+--------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------+
| attached_servers | ['f72d7b2e-12f4-4857-89d5-177bc845e15f'] |
| attachment_ids | ['e59dbb3c-0f5b-443d-84c4-c81519c03794'] |
| availability_zone | rg2-test-control003.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-01-04T03:36:46.000000 |
| description | None |
| encrypted | False |
| id | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 |
| metadata | attached_mode : rw |
| migration_status | migrating | #状态
| multiattach | False |
| name | test-volume-vdb |
| os-vol-host-attr:host | rg2-test-control003.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | migrating |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | dd94c9e5d29048b68b56131a0d6ea2ba |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2022-01-04T03:45:37.000000 |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
| volume_type | lvm |
+--------------------------------+-----------------------------------------------------+
+--------------------------------+-----------------------------------------------------+
[root@rg2-test-control001 ~]# cinder show d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
+--------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------+
| attached_servers | ['f72d7b2e-12f4-4857-89d5-177bc845e15f'] |
| attachment_ids | ['0903c19d-c1e9-4a8e-a376-46eeb8eabc15'] |
| availability_zone | rg2-test-control003.ostack.hfb3.iflytek.net |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-01-04T03:36:46.000000 |
| description | None |
| encrypted | False |
| id | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 |
| metadata | attached_mode : rw |
| migration_status | success | #状态
| multiattach | False |
| name | test-volume-vdb |
| os-vol-host-attr:host | rg2-test-control001.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | success |
| os-vol-mig-status-attr:name_id | 44d67e42-f17d-4114-8b64-569f07b2d9e6 | #新盘
| os-vol-tenant-attr:tenant_id | dd94c9e5d29048b68b56131a0d6ea2ba |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2022-01-04T03:47:44.000000 |
| user_id | 96f12fa88f0e4bb29123dd33021307ad |
| volume_type | lvm |
+--------------------------------+-----------------------------------------------------+
采用的最终方案
经过验证及当期场景，采用如下方案：
4、虚机(本地盘)+数据盘 三者在同一计算节点。(在方案1的基础上调整)
#环境准备：
# ① 需要单独为nova准备一个lv，用来存放虚机系统盘及数据,可以单独VG，也可以和数据盘混用一个VG
# ② 挂载到指定节点 /var/lib/docker/volumes/nova_compute/_data
# ③ 关于数据盘问题: 单独新建一个vg，用来存储数据盘，每一个数据盘都是一个独立的LV，迁移后会生成新的LV。
# 参考该字段value: os-vol-mig-status-attr:name_id
[root@rg2-test-control003 ~]# lvcreate -L 1024G -n lv_nova_instances cinder-volumes1
Logical volume "lv_nova_instances" created.
[root@rg2-test-control003 ~]# mkfs.xfs /dev/cinder-volumes1/lv_nova_instances
meta-data=/dev/cinder-volumes1/lv_nova_instances isize=512 agcount=4, agsize=67108864 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=268435456, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=131072, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@rg2-test-control003 nova_compute]# mv _data _data1
[root@rg2-test-control003 nova_compute]# mount /dev/cinder-volumes1/lv_nova_instances \
/var/lib/docker/volumes/nova_compute/_data
[root@rg2-test-control003 nova_compute]# mv _data1/* _data/
[root@rg2-test-control003 nova_compute]# docker restart nova_libvirt nova_compute
(nova-compute)[root@rg2-test-control003 instances]# ll
total 4
drwxr-xr-x 2 nova nova 73 Jan 4 15:22 1acd4814-314c-4482-803a-8d74fbd3d5f2
drwxr-xr-x. 2 nova nova 6 Dec 21 00:49 a01ffde2-4131-450d-a83b-09ced05035c0
drwxr-xr-x. 2 nova nova 54 Jan 4 10:55 _base
drwxr-xr-x 2 nova nova 44 Dec 23 10:14 cc932408-2806-4798-85f4-b0946a207c68
-rw-r--r--. 1 nova nova 65 Jan 4 14:58 compute_nodes
drwxr-xr-x 2 nova nova 61 Dec 23 18:01 f29a8284-f4ad-473d-90b6-6b3f0c73869f
drwxr-xr-x 2 nova nova 73 Jan 4 15:22 f34b4532-d07b-40fa-83fe-91972bcf9f5c
drwxr-xr-x 2 nova nova 73 Jan 4 11:40 f72d7b2e-12f4-4857-89d5-177bc845e15f
drwxr-xr-x. 2 nova nova 93 Jan 4 10:54 locks
(nova-compute)[root@rg2-test-control003 instances]# cd 1acd4814-314c-4482-803a-8d74fbd3d5f2
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# ls
console.log disk disk.config disk.info
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# du -sh *
0 console.log
26M disk
460K disk.config
4.0K disk.info
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 223G 207G 17G 93% /
tmpfs tmpfs 63G 0 63G 0% /sys/fs/cgroup
devtmpfs devtmpfs 63G 0 63G 0% /dev
tmpfs tmpfs 63G 0 63G 0% /dev/shm
tmpfs tmpfs 63G 486M 63G 1% /run
tmpfs tmpfs 13G 0 13G 0% /run/user/0
/dev/sda2 xfs 223G 207G 17G 93% /etc/iscsi
/dev/mapper/cinder--volumes1-lv_nova_instances xfs 1.0T 11G 1014G 1% /var/lib/nova #存储侧
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]#
[root@rg2-test-control001 ~]# docker ps -a | grep nova
276b84b58fd1 mmwei.com/lokolla/centos-source-nova-compute:stable-15.0.6-ccr-1.2.1 "kolla_start" 2 days ago Up 2 days nova_compute
ad16ad4c615d mmwei.com/lokolla/centos-source-nova-libvirt:15.0.6-ccr-1.1.4 "kolla_start" 3 days ago Up 3 days nova_libvirt
6dcdb7423867 mmwei.com/lokolla/centos-source-nova-novncproxy:stable-15.0.6-ccr-1.1.4 "kolla_start" 4 days ago Up 4 days nova_novncproxy
1b1c7451ff36 mmwei.com/lokolla/centos-source-nova-consoleauth:15.0.6-ccr-1.1.2 "kolla_start" 4 days ago Up 4 days nova_consoleauth
eb19b7296467 mmwei.com/lokolla/centos-source-nova-placement-api:15.0.6-ccr-1.1.2 "kolla_start" 4 days ago Up 4 days placement_api
b43c7de31f2e mmwei.com/lokolla/centos-source-nova-scheduler:15.0.6-ccr-1.1.2 "kolla_start" 4 days ago Up 4 days nova_scheduler
8fdd76b9e13a mmwei.com/lokolla/centos-source-nova-conductor:stable-15.0.6-ccr-1.1.10 "kolla_start" 4 days ago Up 4 days nova_conductor
6609b35806ae mmwei.com/lokolla/centos-source-nova-api:stable-15.0.6-ccr-1.2.1 "kolla_start" 4 days ago Up 4 days nova_api
abad2cdb9be2 mmwei.com/lokolla/centos-source-nova-ssh:15.0.6-ccr-1.1.4 "kolla_start" 4 days ago Up 4 days nova_ssh
[root@rg2-test-control001 ~]#
集群环境
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 223.1G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 222.6G 0 part /
sdb 8:16 0 1.8T 0 disk
sdc 8:32 0 1.8T 0 disk
sdd 8:48 0 2.2T 0 disk
sde 8:64 0 2.2T 0 disk
sdf 8:80 0 2.2T 0 disk
sdg 8:96 0 2.2T 0 disk
sdh 8:112 0 2.2T 0 disk
sdi 8:128 0 2.2T 0 disk
sdj 8:144 0 2.2T 0 disk
sdk 8:160 0 2.2T 0 disk
两块ssd做raid1
8块sata 做raid10
其中 raid10 做1个VG，划分出一个1T的lvm 用来存放虚机系统盘数据，其余做数据盘
到了这里，关于基于本地存储LVM新建虚机方案的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
