Class 3:RTMPose关键点检测
主讲人:张子豪(同济子豪兄)https://space.bilibili.com/1900783
课程地址:RTMPose关键点检测-安装MMDetection和MMPose_哔哩哔哩_bilibili
MMPose主页:https://github.com/open-mmlab/mmpose
教程地址:TommyZihao/MMPose_Tutorials: Jupyter notebook tutorials for mmpose (github.com)
概述
以三角板关键点检测场景为例,结合OpenMMLab开源目标检测算法库MMDetection、开源关键点检测算法库MMPose、开源模型部署算法库MMDeploy,全面讲解项目全流程:
- 数据集:Labelme标注数据集、整理标注格式至MS COCO
- 目标检测:分别训练
Faster R CNN
和RTMDet-Tiny
目标检测模型、训练日志可视化、测试集评估、对图像、摄像头画面预测 - 关键点检测:训练
RTMPose-S
关键点检测模型、训练日志可视化、测试集上评估、分别对“图像、视频、摄像头画面”预测 - 模型终端部署:转ONNX格式,终端推理
视频教程合集:https://space.bilibili.com/1900783/channel/collectiondetail?sid=1316981
安装相关库
为了方便使用者快速上手 MMPose,这次课程有着丰富的示例代码脚本,以及详细的技术文档,包括安装、数据集准备、使用教程、常见问题解答等。通过这些示例和文档,我们可以迅速了解 MMPose 的主要功能和特性,以及在具体场景中的使用方法。
根据:MMPose_Tutorials/【A1】安装MMPose.ipynb at main · TommyZihao/MMPose_Tutorials · GitHub
安装MMDetection
按照顺序逐行运行下面的代码,即可安装配置 MMDetection 环境
https://github.com/TommyZihao/MMPose_Tutorials/blob/main/2023/0524/%E3%80%90A2%E3%80%91%E5%AE%89%E8%A3%85MMDetection.ipynb
预训练模型推理
模型库预训练模型
- 目标检测模型
MMDetection模型库:https://github.com/open-mmlab/mmdetection/blob/master/docs/en/model_zoo.md
demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py
https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
- MMPose人体姿态估计模型
configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py
https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth
- RTMPose人体姿态估计模型
RTMPose主页:https://github.com/open-mmlab/mmpose/tree/dev-1.x/projects/rtmpose
RTMPose-S
projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py
https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.pth
RTMPose-L
projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py
https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-384x288-97d6cb0f_20230228.pth
预测单张图像
In [2]:
# HRNet
!python demo/topdown_demo_with_mmdet.py \
demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth \
--input data/test/multi-person.jpeg \
--output-root outputs/B1_HRNet_1 \
--device cuda:0 \
--bbox-thr 0.5 \
--kpt-thr 0.2 \
--nms-thr 0.3 \
--radius 8 \
--thickness 4 \
--draw-bbox \
--draw-heatmap \
--show-kpt-idx
预测视频:
直接将--input
换成视频路径即可
MMPose官方可视化工具visualizer
img = mmcv.imread(img_path)
img = mmcv.imconvert(img, 'bgr', 'rgb')
img_output = visualizer.add_datasample(
'result',
img,
data_sample=data_samples,
draw_gt=False,
draw_heatmap=True,
draw_bbox=True,
show_kpt_idx=True,
show=False,
wait_time=0,
out_file='outputs/B2.jpg'
)
展示可视化效果
MMDetection三角板目标检测
下载三角板关键点检测数据集
下载用于测试的图像和视频
!mkdir data/test_triangle
# 图像-30度直角三角板,拍摄:同济子豪兄、田文博
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_1.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_2.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_3.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_4.jpg -P data/test_triangle
# 视频-30度直角三角板,拍摄:同济子豪兄,田文博
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_6.mp4 -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_7.mp4 -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_9.mp4 -P data/test_triangle
查看数据集中的图片
# from PIL import Image
# Image.open('data/Triangle_215_Keypoint_coco/images/DSC_0373.jpg')
三角板目标检测-训练
进入mmdetection主目录
import os
os.chdir('mmdetection')
目标检测算法:Faster R CNN
# 建议在命令行中运行
!python tools/train.py data/faster_r_cnn_triangle.py
目标检测算法:RTMDet
# 建议在命令行中运行
!python tools/train.py data/rtmdet_tiny_triangle.py
在work_dirs
目录下,查看训练日志和训练得到的模型权重文件
测试集上评估模型精度
!python tools/test.py data/rtmdet_tiny_triangle.py \
work_dirs/rtmdet_tiny_triangle/epoch_200.pth
训练集损失函数
df_train.columns
Index(['lr', 'data_time', 'loss', 'loss_rpn_cls', 'loss_rpn_bbox', 'loss_cls',
'acc', 'loss_bbox', 'time', 'epoch', 'memory', 'step'],
dtype='object')
metrics = ['loss', 'loss_bbox', 'loss_cls', 'loss_rpn_cls', 'loss_rpn_bbox']
plt.figure(figsize=(16, 8))
x = df_train['step']
for y in metrics:
plt.plot(x, df_train[y], label=y, **get_line_arg())
plt.tick_params(labelsize=20)
plt.xlabel('step', fontsize=20)
plt.ylabel('loss', fontsize=20)
plt.title('训练集损失函数', fontsize=25)
plt.savefig('训练集损失函数.pdf', dpi=120, bbox_inches='tight')
plt.legend(fontsize=20)
plt.show()
训练集准确率
metrics = ['acc']
plt.figure(figsize=(16, 8))
x = df_train['step']
for y in metrics:
plt.plot(x, df_train[y], label=y, **get_line_arg())
plt.tick_params(labelsize=20)
plt.xlabel('step', fontsize=20)
plt.ylabel('loss', fontsize=20)
plt.title('训练集准确率', fontsize=25)
plt.savefig('训练集准确率.pdf', dpi=120, bbox_inches='tight')
plt.legend(fontsize=20)
plt.show()
测试集评估指标-MS COCO Metric
df_test.columns
Index(['coco/bbox_mAP', 'coco/bbox_mAP_50', 'coco/bbox_mAP_75',
'coco/bbox_mAP_s', 'coco/bbox_mAP_m', 'coco/bbox_mAP_l',
'pascal_voc/mAP', 'pascal_voc/AP50', 'data_time', 'time', 'step'],
dtype='object')
metrics = ['coco/bbox_mAP', 'coco/bbox_mAP_50', 'coco/bbox_mAP_75', 'coco/bbox_mAP_s', 'coco/bbox_mAP_m', 'coco/bbox_mAP_l']
plt.figure(figsize=(16, 8))
x = df_test['step']
for y in metrics:
plt.plot(x, df_test[y], label=y, **get_line_arg())
plt.tick_params(labelsize=20)
# plt.ylim([0, 100])
plt.xlabel('Epoch', fontsize=20)
plt.ylabel(y, fontsize=20)
plt.title('测试集评估指标', fontsize=25)
plt.savefig('测试集分类评估指标.pdf', dpi=120, bbox_inches='tight')
plt.legend(fontsize=20)
plt.show()
测试集评估指标-PASCAL VOC Metric
metrics = ['pascal_voc/mAP', 'pascal_voc/AP50']
plt.figure(figsize=(16, 8))
x = df_test['step']
for y in metrics:
plt.plot(x, df_test[y], label=y, **get_line_arg())
plt.tick_params(labelsize=20)
# plt.ylim([0, 100])
plt.xlabel('Epoch', fontsize=20)
plt.ylabel(y, fontsize=20)
plt.title('测试集评估指标', fontsize=25)
plt.savefig('测试集分类评估指标.pdf', dpi=120, bbox_inches='tight')
plt.legend(fontsize=20)
plt.show()
三角板目标检测-预测
三角板目标检测-模型权重文件精简转换
经过精简转换之后,模型.pth权重文件大小缩小为原来的一半以上,但不影响推理结果和推理速度
进入 mmdetection 主目录
import os
os.chdir('mmdetection')
模型轻量化转换
# Faster R CNN
!python tools/model_converters/publish_model.py \
work_dirs/faster_r_cnn_triangle/epoch_50.pth \
checkpoint/faster_r_cnn_triangle_epoch_50_202305120846.pth
05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `message_hub` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `optimizer` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `param_schedulers` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:50 - mmengine - [4m[37mINFO[0m - The published model is saved at checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth.
# RTMDet-tiny
!python tools/model_converters/publish_model.py \
work_dirs/rtmdet_tiny_triangle/epoch_200.pth \
checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847.pth
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `message_hub` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `optimizer` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `param_schedulers` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `ema_state_dict` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - The published model is saved at checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth.
模型权重文件保存在checkpoint
目录
进入mmdetection主目录
import os
os.chdir('mmdetection')
目标检测预测-单张图像
# Faster R CNN
!python demo/image_demo.py \
data/test_triangle/triangle_3.jpg \
data/faster_r_cnn_triangle.py \
--weights checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth \
--out-dir outputs/E2_faster_r_cnn \
--device cuda:0 \
--pred-score-thr 0.3
Loads checkpoint by local backend from path: checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth
05/18 10:35:17 - mmengine - [5m[4m[33mWARNING[0m - Failed to search registry with scope "mmdet" in the "function" registry tree. As a workaround, the current "function" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/18 10:35:17 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[2KInference [91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m [36m [0m
[?25hresults have been saved at outputs/E2_faster_r_cnn
# RTMDet
!python demo/image_demo.py \
data/test_triangle/triangle_3.jpg \
data/rtmdet_tiny_triangle.py \
--weights checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth \
--out-dir outputs/E2_rtmdet \
--device cuda:0 \
--pred-score-thr 0.3
Loads checkpoint by local backend from path: checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth
05/18 10:35:32 - mmengine - [5m[4m[33mWARNING[0m - Failed to search registry with scope "mmdet" in the "function" registry tree. As a workaround, the current "function" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/18 10:35:32 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[2K/environment/miniconda3/lib/python3.7/site-packages/torch/functional.py:445: [0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m [36m [0m
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass
the indexing argument. (Triggered internally at
../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[2KInference [91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m [36m [0m
[?25hresults have been saved at outputs/E2_rtmdet
目标检测预测-视频
# Faster R CNN
!python demo/video_demo.py \
data/test_triangle/triangle_9.mp4 \
data/faster_r_cnn_triangle.py \
checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth \
--device cuda:0 \
--score-thr 0.96 \
--out outputs/E2_out_video_faster_r_cnn.mp4
Loads checkpoint by local backend from path: checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth
05/18 10:35:47 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[>>>>>>>>>>>>> ] 142/318, 3.4 task/s, elapsed: 42s, ETA: 52s/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:759: UserWarning: Warning: The bbox is out of bounds, the drawn bbox may not be in the image
' the drawn bbox may not be in the image', UserWarning)
/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:830: UserWarning: Warning: The polygon is out of bounds, the drawn polygon may not be in the image
' the drawn polygon may not be in the image', UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 3.4 task/s, elapsed: 93s, ETA: 0s
# RTMDet
!python demo/video_demo.py \
data/test_triangle/triangle_9.mp4 \
data/rtmdet_tiny_triangle.py \
checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth \
--device cuda:0 \
--score-thr 0.6 \
--out outputs/E2_out_video_rtmdet.mp4
Loads checkpoint by local backend from path: checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth
05/18 10:37:30 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[ ] 0/318, elapsed: 0s, ETA:/environment/miniconda3/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[>>>>>>>>>>>>> ] 142/318, 4.0 task/s, elapsed: 36s, ETA: 45s/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:759: UserWarning: Warning: The bbox is out of bounds, the drawn bbox may not be in the image
' the drawn bbox may not be in the image', UserWarning)
/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:830: UserWarning: Warning: The polygon is out of bounds, the drawn polygon may not be in the image
' the drawn polygon may not be in the image', UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 4.0 task/s, elapsed: 80s, ETA: 0s
三角板关键点检测
过程和detection相似,这选取一些不同的地方:文章来源:https://www.toymoban.com/news/detail-712629.html
训练RTMPose
同济子豪兄 2023-4-3 5-23文章来源地址https://www.toymoban.com/news/detail-712629.html
进入 mmpose 主目录
import os
os.chdir('mmpose')
训练(建议在命令行中运行)
!python tools/train.py data/rtmpose-s-triangle.py
测试集上评估模型精度
!python tools/test.py data/rtmpose-s-triangle.py \
work_dirs/rtmpose-s-triangle/epoch_300.pth
in the image’, UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 4.0 task/s, elapsed: 80s, ETA: 0s
三角板关键点检测
过程和detection相似,这选取一些不同的地方:
训练RTMPose
同济子豪兄 2023-4-3 5-23
进入 mmpose 主目录
import os
os.chdir('mmpose')
训练(建议在命令行中运行)
!python tools/train.py data/rtmpose-s-triangle.py
测试集上评估模型精度
!python tools/test.py data/rtmpose-s-triangle.py \
work_dirs/rtmpose-s-triangle/epoch_300.pth
到了这里,关于OpenMMLab-AI实战营第二期-课程笔记-Class 3:RTMPose关键点检测的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!