一、基础概念
VA-API
Video Acceleration API 是一组开源应用API接口,赋能应用(比如VLC播放器、GStreamer等)使用hardware video acceleration(一般是GPU提供硬件视频加速功能),VA-API主要由开源库libva和一些硬件驱动(通常是GPU厂商提供)来实现的。
VA-API视频编解码接口独立于平台和窗口系统的,其主要使用场景目标是类Unix系统(如Linux、FreeBSD、Solaris等)和Andriod上X window系统中的直接渲染基础设置(DRI)。加速处理一般包括视频解码、视频编码、子图片混合、渲染。
VA-API最初由intel为其GPU特定功能开发的,现在已经扩展到其他硬件厂商平台。
VA-API如果存在的话,对于某些应用来说可能默认就使用它,比如MPV。
VA-API,对于Debian系统中,视频编解码硬件加速驱动:
-
对于nouveau和大部分的AMD驱动,VA-API通过安装mesa-va-drivers来支持。
-
对于intel,要分免费和付费两种,免费的一般只能decode解码,付费的可支持编码。
-
对于Gen 8+ intel硬件,免费的VA-API可通过安装intel-media-va-driver包,付费包是intel-media-va-driver-non-free。
-
对于比较旧的intel硬件,免费包是i965-va-driver,付费包i965-va-driver-shaders
-
可以通过环境变量LIBVA_DRIVER_NAME来指定选择使用哪个driver驱动,比如LIBVA_DRIVER_NAME=i965(指定使用i965-va-driver),LIBVA_DRIVER_NAME=iHD(指定使用intel-media-va-driver驱动)。
我们可以看下我这里ubuntu中的VA-API各种driver安装的驱动
root@p:~# dpkg -L mesa-va-drivers
/usr/lib/x86_64-linux-gnu/dri/nouveau_drv_video.so
/usr/lib/x86_64-linux-gnu/dri/r600_drv_video.so
/usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
root@pc:~# dpkg -L intel-media-va-driver
/usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
root@pc:~# dpkg -L i965-va-driver
/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
VDPAU
Video Decode and Presentation API for Unix 是基于开源库libvdpau开发的免费API库,被Nvidia支持,其目标是实现device drivers如Nvidia GeForce driver、nouveau、amdgpu等,给VLC等终端应用提供视频硬件加速处理功能功能。
VDPAU的目标平台是类Unix系统,包括Linux、FreeBSD、Solaris。
VDPAU相对于VA-API来说,有更多的局限
Debian系统上,硬件加速驱动:
-
对于AMD驱动(radeon和amdgpu),以及支持Nvidia的开源驱动nouveau来说,可以通过安装vdpau-driver-all来支持实现,同时,该驱动也能支持是OpenGL/VA-API后端的intel GPUs,不过不能支持所有的intel设备(对于intel设备最好还是使用VA-API)。
-
对于proprietary nvidia硬件来说,安装nvidia-vdpau-driver来支持。
root@pc:~# dpkg -L mesa-vdpau-drivers
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/vdpau
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nouveau.so.1.0.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r300.so.1.0.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r600.so.1.0.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_radeonsi.so.1.0.0
/usr/share
/usr/share/bug
/usr/share/bug/mesa-vdpau-drivers
/usr/share/bug/mesa-vdpau-drivers/control
/usr/share/bug/mesa-vdpau-drivers/script
/usr/share/doc
/usr/share/doc/mesa-vdpau-drivers
/usr/share/doc/mesa-vdpau-drivers/changelog.Debian.gz
/usr/share/doc/mesa-vdpau-drivers/copyright
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nouveau.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nouveau.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nouveau.so.1.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r300.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r300.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r300.so.1.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r600.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r600.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_r600.so.1.0
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_radeonsi.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_radeonsi.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_radeonsi.so.1.0
root@pc:~# dpkg -L vdpau-driver-all
/.
/usr
/usr/share
/usr/share/bug
/usr/share/bug/vdpau-driver-all
/usr/share/doc
/usr/share/doc/vdpau-driver-all
/usr/share/doc/vdpau-driver-all/changelog.Debian.gz
/usr/share/doc/vdpau-driver-all/copyright
NVENC/NVDEC
Debian系统中:
-
NVDEC可以通过安装libnvcuvid1来支持。
-
NVENC可以通过安装libnvidia-encode1来支持。
VA-API/VDPAU/NVENC对比
在Linux平台下,video decoding and encoding的hardware acceleration主要有三种API接口: VA-API, VDPAU, NVEND/NVDEC
Linux平台APIs | 硬件厂商支持 | 软件应用支持 | 主要缺陷 |
---|---|---|---|
VA-API | Intel AMD Nvidia(只能借助开源驱动nouveau支持) |
kodi,VLC,MPV chromium,firefox等 |
缺乏对Nvidia的专有驱动的支持(lacking any support in the proprietary Nvidia drivers) |
VDPAU | fully supported in AMD Nvidia(专有驱动或者开源驱动nouveau都支持) |
大部分desktop应用都支持,如kodi,VLC,MPV等,但在chromium、firefox中不支持 |
对intel的支持不完善 不支持浏览器上的web video acceleration |
NVENC/NVDEC | Nvidia专有的API | 只能支持较少的应用 支持ffmpeg和OBS studio的编码 支持ffmpeg和MPV的解码 |
由于其proprietary(专用),不能跨board支持,导致只能支持较少的软硬件 |
DXVA
DirectX Video Acceleration 是微软平台和Xbox 360平台用于视频解码硬件加速的一组Microsoft API。DXVA2.0扩展了更多的操作,比如视频捕捉和处理的硬件加速。
Windows平台API | ||
---|---|---|
DXVA/DXVA2 |
VirGL
VirGL is a virtual 3D GPU for use inside QEMU virtual machines, that allows the guest operating system to use the capabilities of the host GPU to accelerate 3D rendering. The plan is to have a guest GPU that is fully independent of the host GPU.
virglrender
virglrenderer是一个开源项目virgl3d提供的开源库,它的主要功能是针对虚拟化场景,为QEMU提供一个具有3D图形处理的显卡,其使用方式就是为QEMU提供一组3D图形处理的接口。QEMU通过调用virglrenderer的库接口实现主机侧的3D图形加速处理。
VM之QEMU中3D加速调用栈:
virtio-gpu
网上找的一张图解释的很好
mesa
The Mesa project began as an open-source implementation of the OpenGL specification - a system for rendering interactive 3D graphics.
Mesa是opengl标准的一种开源实现。OpenGL API是定义了一个跨编程语言、 跨平台的应用程序接口(API)的规范, 它用于生成2D和3D图像, 而它仅仅是定义了一种API, 并没有任何实现细节. 而OpenGL API的具体实现有很多, 主要分为开源实现和闭源实现, 闭源实现如各大GPU厂商自己实现的闭源OpenGL图形库, 例如AMD显卡的Catalyst闭源驱动; 而开源实现便是Mesa3D, 它是由Brian Paul在1993年8月开始开发的一个实现了OpenGL API的开源图形库. 它目前隶属于freedesktop.org, 广泛运用在Liunx, BSD等操作系统。
主要由mesa主模块、gallium模块、egl模块、glsl模块和glx等模块组成。
Over the years the project has grown to implement more graphics APIs, including OpenGL ES, OpenCL, OpenMAX, VDPAU, VA-API, Vulkan and EGL。
二、virglrender支持h264/h265编解码
virglrender项目开源仓库地址:https://gitlab.freedesktop.org/virgl/virglrenderer.git
其中,video的支持需要安装libva-devel库,比如virgl_video.c中需要#include <va/va.h>
,而/usr/include/va/va.h
是由libva-devel安装包提供的:
[root@244:libva#] rpm -ql libva-devel
/usr/include/va
/usr/include/va/va.h
/usr/include/va/va_backend.h
...
其中virglrenderer中已经添加了对h264/h265的编解码vaapi加速支持,主要提交代码修改点这里简要分析下:
mesa改动
1. 增加virgl_video_hw.h
主要将原来的src/gallium/drivers/virgl/virgl_video.h
中的部分代码移动到src/virtio/virtio-gpu/virgl_video_hw.h
中来管理。
这样,原来的virgl_video.h中就只剩定义两个东西:
- virgl_video_buffer:用于存储原始YUV格式的数据buffer
- virgl_video_codec:定义encoder或者decoder
virgl_video_hw.h中主要定义用于guest和host之间交互的数据结构:
- 4字节对其
- virgl_picture_desc等相关数据结构主要定义sequence parameters、picture parameters、slice parameters以及用于编解码的context信息等,video backend需要这些参数去重构VA-API calls
2. 保存intra_idr_period数据在context
首先结构体struct pipe_h264_enc_picture_desc中增加该元素数据intra_idr_period
VAEncSequenceParameterBufferH264该结构体是libva库中va_enc_h264.h中定义的。
3. 传递max_references至backend
encoding SPS header是,max_references也是一个重要的sequence parameter值,需要传递。
主要是修改src/gallium/drivers/virgl/virgl_encode.c
中,virgl_encode_create_video_codec()函数中判断host_feature_check_version是否大于等于14,是的话就要传递max_references,通过virgl_encoder_write_dword()来写。
4. 实现编码框架并支持h264编码
src/gallium/drivers/virgl/virgl_video.c
中增加一个函数virgl_encode_encode_bitstream
,主要是给传递指令和资源给backend。
主要修改是virgl_video.c中,其中需要的数据结构增加在virgl_video_hw.h中
virgl_video_hw.h增加了哪些数据结构?
struct virgl_enc_quality_modes {};
enum virgl_video_encode_stat {};
struct virgl_video_encode_feedback {};
//下面是h264相关的
struct virgl_h264_enc_seq_param {};
struct virgl_h264_enc_rate_control {};
struct virgl_h264_enc_motion_estimation {};
struct virgl_h264_enc_pic_control {};
struct virgl_h264_slice_descriptor {};
struct virgl_h264_enc_picture_desc {};
virgl_video.c中增加的函数主要有,框架相关的
static int fill_enc_picture_desc(const struct pipe_picture_desc *desc,
union virgl_picture_desc *vdsc)
static void virgl_video_encode_bitstream(struct pipe_video_codec *codec,
struct pipe_video_buffer *source,
struct pipe_resource *target,
void **feedback)
h264相关的
static int fill_h264_enc_picture_desc(const struct pipe_picture_desc *desc,
union virgl_picture_desc *vdsc)
5. 支持h265编码
virgl_video_hw.h中增加的主要数据结构有:
struct virgl_h265_enc_seq_param {};
struct virgl_h265_enc_pic_param {};
struct virgl_h265_enc_slice_param {};
struct virgl_h265_enc_rate_control {};
struct virgl_h265_slice_descriptor {};
struct virgl_h265_enc_picture_desc {};
virgl_video.c中增加的函数主要
static int fill_h265_enc_picture_desc(const struct pipe_picture_desc *desc,
union virgl_picture_desc *vdsc)
static int fill_mpeg4_picture_desc(const struct pipe_picture_desc *desc,
union virgl_picture_desc *vdsc)
{
switch (u_reduce_video_profile(desc->profile)) {
case PIPE_VIDEO_FORMAT_MPEG4_AVC:
return fill_h264_enc_picture_desc(desc, vdsc); //h264入口
case PIPE_VIDEO_FORMAT_HEVC:
return fill_h265_enc_picture_desc(desc, vdsc); //这里是h265入口
default:
return -1;
}
virglrender改动
1. 增加virgl_video_hw.h
类似mesa中一样,将部分数据结构单独领出来放到virgl_video_hw.h中,这样
virgl_video.h中主要就剩余定义
- virgl_video_buffer:存储原始YUV数据的buffer,在VA-API中一般和surface处理一起出现
- virgl_video_codec:代表一个编码器或解码器,在VA-API中一般随context处理出现
virgl_video_hw.h中定义数据结构主要有:
struct virgl_base_picture_desc {};
struct virgl_h264_sps {}; //h264 sequence parameter集
struct virgl_h264_pps {}; //h264 picture parameter集
struct virgl_h264_picture_desc {};
struct virgl_h265_sps {};
struct virgl_h265_pps {};
struct virgl_h265_picture_desc {};
struct virgl_mpeg4_picture_desc {};
union virgl_picture_desc {};
2. 一些优化代码
为了更好的支持encode,他们对代码进行了优化修改,比如:
- struct virgl_video_create_buffer_args结构体中增加opaque数据指针
- 重新实现了export_video_dma_buf()接口使代码看起来更简洁
- virgl_video_init()的cb参数改成了结构体,这样可以传递更多而callback函数
- 在virgl_video_end_frame()中总是会调用vaSyncSurface(),原来仅是在decode context中调用
- 增加level、max_references等参数,在encoding SPS headers中需要使用
3. 支持h264编码
virgl_video_hw.h中增加部分数据结构体:
struct virgl_enc_quality_modes {};
struct virgl_h264_enc_seq_param {};
struct virgl_h264_enc_rate_control {};
struct virgl_h264_enc_motion_estimation {};
struct virgl_h264_enc_pic_control {};
struct virgl_h264_slice_descriptor {};
struct virgl_h264_enc_picture_desc {};
enum virgl_video_encode_stat {};
struct virgl_video_encode_feedback {};
主要修改virgl_video.c
virgl_video.c中实现了一个通用编解码器的接口,主要基于VA-API实现。
-
virgl_video_buffer的实现,用于存放原始YUV数据,其实对VASurface的一层包装
-
virgl_video_codec实现,主要是对VAContext的包装,主要提供功能有:
-
virgl_video_begin_frame()
该函数会调用vaBeginPicture()来准备编码或解码,对于encoding,它还需要guest端上传raw picture数据至本地VASurface
-
virgl_video_decode_bitstream()
根据picture desc描述的信息,构造decoding相关的VABuffers,然后调用vaRenderPicture()来decoding
-
virgl_video_encode_bitstream()
根据picture desc描述信息来构造encoding相关的VABuffer,然后调用vaRenderPicture()来encoding
-
virg_video_end_frame()
调用vaEndPicure()来结束编码或解码;对于decode,会将raw picture data从VASurface中传回guest端;对于encode,会将VACodedBuffer中的编码结果数据传给guest端
-
新增的主要函数:
static void encode_upload_picture(struct virgl_video_codec *codec,
struct virgl_video_buffer *buffer)
static void encode_completed(struct virgl_video_codec *codec,
struct virgl_video_buffer *buffer)
static VASurfaceID get_enc_ref_pic(struct virgl_video_codec *codec,
uint32_t frame_num)
static void h264_fill_enc_picture_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc,
VAEncPictureParameterBufferH264 *param)
static void h264_fill_enc_slice_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc,
VAEncSliceParameterBufferH264 *param)
static void h264_fill_enc_seq_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc,
VAEncSequenceParameterBufferH264 *param)
static void h264_fill_enc_misc_param_frame_rate(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc,
VAEncMiscParameterFrameRate *param)
static int h264_encode_render_sequence(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc)
static int h264_encode_render_picture(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc)
static int h264_encode_render_slice(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc)
static int h264_encode_bitstream(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h264_enc_picture_desc *desc)
int virgl_video_encode_bitstream(struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const union virgl_picture_desc *desc)
vrend_decode.c增加了decode相关函数
static int vrend_decode_encode_bitstream(struct vrend_context *ctx,
const uint32_t *buf,
uint32_t length)
vrend_video.c
static void vrend_video_enocde_upload_picture(
struct virgl_video_codec *codec,
const struct virgl_video_dma_buf *dmabuf)
static void vrend_video_encode_completed(
struct virgl_video_codec *codec,
const struct virgl_video_dma_buf *src_buf,
const struct virgl_video_dma_buf *ref_buf,
unsigned num_coded_bufs,
const void * const *coded_bufs,
const unsigned *coded_sizes)
int vrend_video_encode_bitstream(struct vrend_video_context *ctx,
uint32_t cdc_handle,
uint32_t src_handle,
uint32_t dest_handle,
uint32_t desc_handle,
uint32_t feed_handle)
4. 增加h265编码支持
主要就修改virgl_video.c和virgl_video_hw.h两个文件
virgl_video_hw.h增加数据结构
struct virgl_h265_enc_seq_param {};
struct virgl_h265_enc_pic_param {};
struct virgl_h265_enc_slice_param {};
struct virgl_h265_enc_rate_control {};
struct virgl_h265_slice_descriptor {};
struct virgl_h265_enc_picture_desc {};
virgl_video.c增加函数
static void h265_init_picture(VAPictureHEVC *pic)
static void h265_fill_enc_seq_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc,
VAEncSequenceParameterBufferHEVC *param)
static void h265_fill_enc_picture_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc,
VAEncPictureParameterBufferHEVC *param)
static void h265_fill_enc_slice_param(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc,
VAEncSliceParameterBufferHEVC *param)
static void h265_fill_enc_misc_param_rate_ctrl(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc,
VAEncMiscParameterRateControl *param)
static void h265_fill_enc_misc_param_frame_rate(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc,
VAEncMiscParameterFrameRate *param)
static int h265_encode_render_sequence(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc)
static int h265_encode_render_picture(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc)
static int h265_encode_render_slice(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc)
static int h265_encode_bitstream(
struct virgl_video_codec *codec,
struct virgl_video_buffer *source,
const struct virgl_h265_enc_picture_desc *desc)
三、虚机环境搭建
在host主机上搭建,我这里主机上有两个显卡:
[root@localhost ubuntu]# lspci | grep VGA
02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
1b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev ff)
主要安装步骤
host准备工作
qemu编译
cd qemu
mkdir build
cd build
../configure --prefix=/root/install --enable-spice --enable-virglrenderer --enable-debug --target-list=x86_64-softmmu
make -j4
virglrenderer编译
cd virglrenderer
meson build --prefix=/root/install -Dvideo=true -Dbackend=ninja -Dbuildtype=debug
cd build
ninja && ninja install
这里的virglrenderer与开源主分支有个区别,添加了个vrend_get_drm_fd()函数,注释掉了暂未支持的AV1处理case:
From 6c403a9544fea082926e3973346cd70275605f3a Mon Sep 17 00:00:00 2001
From: jrg <jrglinuxcn@gmail.com>
Date: Tue, 7 Feb 2023 15:39:15 +0800
Subject: [PATCH 1/2] Add vrend_get_drm_fd() for open /dev/dri/renderD128
Signed-off-by: jrg <jrglinuxcn@gmail.com>
---
src/virgl_video.c | 13 +++++-----
src/virglrenderer.c | 2 +-
src/vrend_renderer.c | 58 +++++++++++++++++++++++++++++++++++++++++---
3 files changed, 62 insertions(+), 11 deletions(-)
diff --git a/src/virgl_video.c b/src/virgl_video.c
index 025ce00..1449ebf 100644
--- a/src/virgl_video.c
+++ b/src/virgl_video.c
@@ -152,8 +152,8 @@ static enum pipe_video_profile pipe_profile_from_va(VAProfile profile)
return PIPE_VIDEO_PROFILE_VP9_PROFILE0;
case VAProfileVP9Profile2:
return PIPE_VIDEO_PROFILE_VP9_PROFILE2;
- case VAProfileAV1Profile0:
- return PIPE_VIDEO_PROFILE_AV1_MAIN;
+ //case VAProfileAV1Profile0:
+ //return PIPE_VIDEO_PROFILE_AV1_MAIN;
case VAProfileNone:
return PIPE_VIDEO_PROFILE_UNKNOWN;
default:
@@ -247,8 +247,8 @@ static VAProfile va_profile_from_pipe(enum pipe_video_profile profile)
return VAProfileVP9Profile0;
case PIPE_VIDEO_PROFILE_VP9_PROFILE2:
return VAProfileVP9Profile2;
- case PIPE_VIDEO_PROFILE_AV1_MAIN:
- return VAProfileAV1Profile0;
+ //case PIPE_VIDEO_PROFILE_AV1_MAIN:
+ //return VAProfileAV1Profile0;
case PIPE_VIDEO_PROFILE_MPEG4_AVC_EXTENDED:
case PIPE_VIDEO_PROFILE_MPEG4_AVC_HIGH10:
case PIPE_VIDEO_PROFILE_MPEG4_AVC_HIGH422:
@@ -557,8 +557,9 @@ int virgl_video_init(int drm_fd,
if (!driver || !strstr(driver, "Mesa Gallium")) {
virgl_log("only supports mesa va drivers now\n");
- virgl_video_destroy();
- return -1;
+ virgl_log("go on ...\n");
+ //virgl_video_destroy();
+ //return -1;
}
callbacks = cbs;
diff --git a/src/virglrenderer.c b/src/virglrenderer.c
index de8b623..6a00198 100644
--- a/src/virglrenderer.c
+++ b/src/virglrenderer.c
@@ -758,7 +758,7 @@ int virgl_renderer_init(void *cookie, int flags, struct virgl_renderer_callbacks
renderer_flags |= VREND_USE_ASYNC_FENCE_CB;
if (flags & VIRGL_RENDERER_USE_EXTERNAL_BLOB)
renderer_flags |= VREND_USE_EXTERNAL_BLOB;
- if (flags & VIRGL_RENDERER_USE_VIDEO)
+ //if (flags & VIRGL_RENDERER_USE_VIDEO)
renderer_flags |= VREND_USE_VIDEO;
ret = vrend_renderer_init(&vrend_cbs, renderer_flags);
diff --git a/src/vrend_renderer.c b/src/vrend_renderer.c
index 7e925ba..6fd78ad 100644
--- a/src/vrend_renderer.c
+++ b/src/vrend_renderer.c
@@ -66,6 +66,10 @@
#ifdef ENABLE_VIDEO
#include <vrend_video.h>
+#include <dirent.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
#endif
/*
@@ -7110,6 +7114,55 @@ static bool use_integer(void) {
return false;
}
+#ifdef ENABLE_VIDEO
+static int vrend_get_drm_fd(const char *rendernode)
+{
+ DIR *dir;
+ struct dirent *e;
+ struct stat st;
+ int r, fd, ret;
+ char p[512];
+
+ if(rendernode) {
+ return open(rendernode, O_RDWR | O_NOCTTY | O_NONBLOCK);
+ }
+
+ dir = opendir("/dev/dri");
+ if(!dir) {
+ return -1;
+ }
+
+ fd = -1;
+ while(( e = readdir(dir))){
+ if (strncmp(e->d_name, "renderD", 7)) {
+ continue;
+ }
+
+ sprintf(p, "dev/dri/%s", e->d_name);
+
+ r = open(p, O_RDWR | O_NOCTTY | O_NONBLOCK);
+ if(r<0){
+ continue;
+ }
+
+ ret = fstat(r, &st);
+ if(ret < 0 || (st.st_mode &S_IFMT) != S_IFCHR){
+ close(r);
+ continue;
+ }
+
+ fd = r;
+ break;
+ }
+
+ closedir(dir);
+ if(fd < 0){
+ return -1;
+ }
+ return fd;
+}
+#endif
+
int vrend_renderer_init(const struct vrend_if_cbs *cbs, uint32_t flags)
{
bool gles;
@@ -7252,10 +7305,7 @@ int vrend_renderer_init(const struct vrend_if_cbs *cbs, uint32_t flags)
#ifdef ENABLE_VIDEO
if (flags & VREND_USE_VIDEO) {
- if (vrend_clicbs->get_drm_fd)
- vrend_video_init(vrend_clicbs->get_drm_fd());
- else
- vrend_printf("video disabled due to missing get_drm_fd\n");
+ vrend_video_init(vrend_get_drm_fd("/dev/dri/renderD128"));
}
#endif
--
2.27.0
创建ubuntu虚机
我这里使用的是ubuntu-22.04.1版本。
最终,待guest中准备工作做完后,ubuntu虚机启动命令如下:
LD_PRELOAD=/root/install/lib64/libvirglrenderer.so.1 \
/root/install/bin/qemu-system-x86_64 \
-enable-kvm \
-smp 4 \
-m 8G \
-device virtio-serial \
-chardev spicevmc,id=vdagent,debug=0,name=vdagent \
-device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \
-drive file=./ubuntu_v01.img,if=virtio \
-spice port=8010,addr=172.17.84.241,disable-ticketing=on \
-cpu host \
-device virtio-gpu-gl \
-display egl-headless,gl=on \
-machine q35,accel=kvm,usb=on \
-device usb-mouse \
-vga none
guest准备工作
libva和libva-utils编译
# libva
./autogen.sh
./configure CFLAGS="-g -O0"
make && make install
# libva-utils
./autogen.sh --prefix=/root/install
make && make install
cp /root/install/bin/vainfo /usr/bin/vainfo
mesa编译
mesa编译的依赖安装:可以用apt build-dep mesa安装大多数依赖包,但还有几个可能需要手动安装,如meson、bindgen等(meson用pip3安装最为便捷)
apt install make gcc g++ bison
apt install libncurses-dev
apt install flex
apt install libssl-dev
apt install libelf-dev
apt install meson
apt install git
apt install pkg-config
apt install cmake
apt install mesa-vdpau-drivers
apt install libvdpau-dev
apt install glslang-dev
apt install glslang-tools
apt install libva-dev
apt-get install python3 python3-pip python3-setuptools python3-wheel ninja-build
pip3 install meson -i https://pypi.tuna.tsinghua.edu.cn/simple
apt install libomxil-bellagio-dev
apt install rust-all
apt install libclc-dev
apt install libzstd-dev
apt install libdrm-dev
apt install llvm-dev
dpkg -i libllvmspirvlib14_14.0.0-5_amd64.deb
dpkg -i libllvmspirvlib-14-dev_14.0.0-5_amd64.deb
apt install libclang-cpp-dev
apt install wayland-protocols
apt install libwayland-egl-backend-dev
apt install libxext-dev
apt install libxfixes-dev
apt install libxcb-glx0-dev
apt install libxcb-shm0-dev
apt install libx11-xcb-dev
apt install libxcb-dri2-0-dev
apt install libxcb-dri3-dev
apt install libxcb-present-dev
apt install libxshmfence-dev
apt install libxxf86vm-dev
apt install libxrandr-dev
apt install python3-codegen
apt install libclang-dev
apt install bindgen
apt install libsensors-dev
mesa编译:
meson -Dgallium-rusticl=true -Dgallium-opencl=standalone -Dllvm=enabled -Drust_std=2021 -Dvalgrind=disabled -Dopencl-spirv=true -Dshader-cache=true -Dvideo-codecs="vc1dec,h264dec,h265dec,h264enc,h265enc" -Dbuildtype=debug build
cd build
ninja && ninja install
vainfo
执行vainfo可能会报错:
-
export LIBVA_DRIVER_NAME=virtio_gpu
-
cp /usr/local/lib/x86_64-linux-gnu/dri/virtio_gpu_drv_video.so /usr/lib/x86_64-linux-gnu/dri/ (可能是拷贝到/usr/local/lib/dri/,要看libva寻找到路径)
root@pc:~# vainfo
Trying display: wayland
libva info: VA-API version 1.17.0
libva info: User environment variable requested driver 'virtio_gpu'
libva info: Trying to open /usr/local/lib/dri/virtio_gpu_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.17.0.pre1)
vainfo: Driver version: Mesa Gallium driver 23.0.0-devel for virgl (Radeon RX 550 Series (POLARIS11, DRM 3.44.0, 4.18.0-...)
vainfo: Supported profile and entrypoints
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileNone : VAEntrypointVideoProc
四、验证h264/h265
gdb调试
host主机debug
host环境配置
host主机上需要配置debuginfo相关,创建CentOS-Debug.repo文件,设置其中enabled=1。并使用debuginfo-install命令安装必要的debug库。
[root@localhost ~]# cat /etc/yum.repos.d/CentOS-Debug.repo
#Debug Info
[debuginfo]
name=CentOS-$releasever - DebugInfo
# CentOS-4
#baseurl=http://debuginfo.centos.org/$releasever/
# CentOS-5
baseurl=http://debuginfo.centos.org/$releasever/$basearch/
gpgcheck=0
enabled=1
# CentOS-4
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-$releasever
# CentOS-5
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
protect=1
priority=1
gdb调试qemu和virglrenderer
在host主机上gdb debug跟踪qemu和virglrender。这里的-vga none参数有时会导致guest虚机启动后没有画面,需要先去除-vga none启动一次虚机,再带-vga none重启虚机。
gdb --args env LD_PRELOAD=/root/install/virglrenderer/build/src/libvirglrenderer.so.1 \
/root/project/qemu/build/qemu-system-x86_64 \
-enable-kvm \
-smp 4 \
-m 8G \
-device virtio-serial \
-chardev spicevmc,id=vdagent,debug=0,name=vdagent \
-device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \
-drive file=/root/ubuntu/ubuntu2204.qcow2,if=virtio \
-spice port=8010,addr=172.17.84.241,disable-ticketing=on \
-cpu host \
-device virtio-gpu-gl \
-display egl-headless,gl=on \
-machine q35,accel=kvm,usb=on \
-device usb-mouse \
-vga none
gdb环境刚起来,首先需要在gdb环境中设置,然后再运行r命令
(gdb) handle SIGUSR1 nostop noprint
Signal Stop Print Pass to program Description
SIGUSR1 No No Yes User defined signal 1
然后我这里设置了一个断点:
(gdb) b virgl_video_decode_bitstream
Breakpoint 1 at 0x7ffff7b83367: file ../src/virgl_video.c, line 2249.
然后我再guest中用vlc播放一个视频,host上virglrenderer中就会运行到断点处.
guest端debug
可以使用ffmpeg或者vlc来对视频进行编解码和播放。
gdb调试ffmpeg和mesa
同样,设置mesa中的断点,比如virgl_video_decode_bitstream(),然后用vlc播放视频,便会执行到断点处。
h264流程
h264编码
测试命令:
ffmpeg -vaapi_device /dev/dri/renderD128 -i ../test.mp4 -vf 'format=nv12,hwupload' -c:v h264_vaapi ../output264.mp4
(virglrenderer)virgl_video.c中h264的encode bitstream大体流程:
h264解码
测试命令,我这里直接用vlc播放刚才编码的h264格式的视频
vlc output264.mp4
经常会出现中间打断vlc播放的话,画面直接卡住(也不是每次必现)
(virglrenderer)virgl_video.c中h264的decode逻辑:
h265流程
h265编码
测试命令:
/ffmpeg -vaapi_device /dev/dri/renderD128 -i ../test.mp4 -vf 'format=nv12,hwupload' -c:v hevc_vaapi ../out265.mp4
(virglrenderer)virgl_video.c中h265_encode_bitstream代码逻辑:
使用如下命令./ffmpeg/ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i test.mp4 -vf 'format=nv12,hwupload' -c:v hevc_vaapi test.h265
进行h265转码,生成test.h265文件。
./ffmpeg/ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i test.mp4 -vf 'format=nv12,hwupload' -c:v hevc_vaapi test.h265
ffmpeg version N-109535-gfcd557a2c2 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
configuration:
libavutil 57. 43.100 / 57. 43.100
libavcodec 59. 56.100 / 59. 56.100
libavformat 59. 34.102 / 59. 34.102
libavdevice 59. 8.101 / 59. 8.101
libavfilter 8. 53.100 / 8. 53.100
libswscale 6. 8.112 / 6. 8.112
libswresample 4. 9.100 / 4. 9.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp41isom
creation_time : 2022-11-25T01:53:42.000000Z
Duration: 00:01:18.88, start: 0.000000, bitrate: 6544 kb/s
Stream #0:0[0x1](und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6346 kb/s, 30.30 fps, 30 tbr, 30302 tbn (default)
Metadata:
creation_time : 2022-11-25T01:53:42.000000Z
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : AVC Coding
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 193 kb/s (default)
Metadata:
creation_time : 2022-11-25T01:53:42.000000Z
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
Press [q] to stop, [?] for help
[hevc_vaapi @ 0x55eb4baeab00] Driver does not advertise encoder features, using guessed defaults.
[hevc_vaapi @ 0x55eb4baeab00] Driver does not advertise encoder block size, using guessed defaults.
[hevc_vaapi @ 0x55eb4baeab00] No quality level set; using default (25).
[hevc_vaapi @ 0x55eb4baeab00] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, hevc, to 'test.h265':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp41isom
encoder : Lavf59.34.102
Stream #0:0(und): Video: hevc (Main), vaapi(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbn (default)
Metadata:
creation_time : 2022-11-25T01:53:42.000000Z
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc59.56.100 hevc_vaapi
frame= 2368 fps= 41 q=-0.0 Lsize= 16339kB time=00:01:18.90 bitrate=1696.5kbits/s dup=0 drop=22 speed=1.35x
video:16339kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
过程中,virglrenderer支持该过程。然后使用ffplay正常播放test.h265,使用ffprobe查看test.h265文件信息
root@pc:~# ./ffmpeg/ffprobe test.h265
Input #0, hevc, from 'test.h265':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1088, 25 fps, 25 tbr, 1200k tbn
h265解码
测试命令:
./ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i ../out265.mp4 ../out265.yuv
(virglrenderer)virgl_video.c中h265的decode逻辑:
yuv编解码
使用rawvideo转码生成yuv
使用rawvideo生成yuv格式文件,该命令下ffmpeg中未使用vaapi,代码执行未经过host主机上的virglrenderer:
./ffmpeg -i ../test.mp4 -c:v rawvideo -pix_fmt yuv420p ../test.yuv
生成的test.yuv有6.9G大小。
用hevc_vaapi来进行转码,生成的test265.mp4是9.6M大小,目前用vlc测试播放是ok的
./ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose \
-f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r:v 30 -i ../test.yuv \
-vf 'format=nv12,hwupload' -c:v hevc_vaapi -b:v 4000k -maxrate 8000k \
-y ../test265.mp4
同样,使用h264_vaapi来进行编码后生成的文件用vlc播放也是没问题的。
使用vaapi转码生成yuv
使用vaapi来转码生成yuv格式,该命令会经过host主机上的virglrenderer:
./ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i ../test.mp4 -pix_fmt yuv420p ../test.yuv
生成的test.yuv有6.9G大小。
如果用解码生成的test.yuv格式再转码生成mp4,生成的test265.mp4是9.4M大小:
./ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose \
-f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r:v 30 -i ../test.yuv \
-vf 'format=nv12,hwupload' -c:v hevc_vaapi -b:v 4000k -maxrate 8000k \
-y ../test265.mp4
经测试,也能正常vlc播放。
测试中发现如果在转yuv的命令中不带"-pix_fmt yuv420p"参数,则后面再转码生成test265.mp4则有39M大小,播放有失真。
MPEG4测试
ffmpeg中应该不支持mpeg4的vaapi加速。
./ffmpeg -i ../test.mp4 -c:v mepg4 ../testmpeg4.mp4
,用该命令生成的视频质量一般。
./ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i ../test.mp4 -c:v mpeg4 ../testmpeg4.mp4
,该命令的话会经过host主机上的virglrenderer,同样生成的视频质量一般。
ffmpeg中支持的vaapi有:
encoders | vaapi surface |
---|---|
H.262 / MPEG-2 part 2 | mpeg2_vaapi |
H.264 / MPEG-4 part 10(AVC) | h264_vaapi |
H.265 / MPEG-H part 2(HEVC) | h265_vaapi |
MJPEG / JPEG | mjpeg_vaapi |
VP8 | vp8_vaapi |
VP9 | vp9_vaapi |
根据代码看,vaapi中也是不支持MPEG4的,for循环中判断PIPE_VIDEO_FROMAT_MEPG4会continue,但是最终还是会返回VA_STATUS_SUCCESS.
VAStatus
vlVaQueryConfigProfiles(VADriverContextP ctx, VAProfile *profile_list, int *num_profiles)
{
struct pipe_screen *pscreen;
enum pipe_video_profile p;
VAProfile vap;
if (!ctx)
return VA_STATUS_ERROR_INVALID_CONTEXT;
*num_profiles = 0;
pscreen = VL_VA_PSCREEN(ctx);
for (p = PIPE_VIDEO_PROFILE_MPEG2_SIMPLE; p <= PIPE_VIDEO_PROFILE_AV1_MAIN; ++p) {
if (u_reduce_video_profile(p) == PIPE_VIDEO_FORMAT_MPEG4 && !debug_get_option_mpeg4())
continue;
if (vl_codec_supported(pscreen, p, false) ||
vl_codec_supported(pscreen, p, true)) {
vap = PipeToProfile(p);
if (vap != VAProfileNone)
profile_list[(*num_profiles)++] = vap;
}
}
/* Support postprocessing through vl_compositor */
profile_list[(*num_profiles)++] = VAProfileNone;
return VA_STATUS_SUCCESS;
}
VC1测试
带-hwaccel vaapi参数将wmv9格式转h264
命令:
./ffmpeg/ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i wmv.wmv -vf 'format=nv12,hwupload' -c:v h264_vaapi test.h264
效果:
其中会报个错:No support for codec wmv3 profile 1。编码过程中会经过virglrenderer(应该是h264经过的)
root@pc:~# ./ffmpeg/ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i wmv.wmv -vf 'format=nv12,hwupload' -c:v h264_vaapi test.h264
[wmv3 @ 0x56333dad8500] Extra data: 8 bits left, value: 0
Input #0, asf, from 'wmv.wmv':
Metadata:
IsVBR : 0
MediaFoundationVersion: 2.112
Duration: 00:01:02.37, start: 0.000000, bitrate: 1032 kb/s
Stream #0:0(eng): Video: wmv3 (Main) (WMV3 / 0x33564D57), yuv420p, 1280x720, 1024 kb/s, SAR 1:1 DAR 16:9, 24 fps, 24 tbr, 1k tbn
File 'test.h264' already exists. Overwrite? [y/N] y
[wmv3 @ 0x56333dae9a00] Extra data: 8 bits left, value: 0
[wmv3 @ 0x56333dae9a00] No support for codec wmv3 profile 1.
[wmv3 @ 0x56333dae9a00] Failed setup for format vaapi: hwaccel initialisation returned error.
Stream mapping:
Stream #0:0 -> #0:0 (wmv3 (native) -> h264 (h264_vaapi))
Press [q] to stop, [?] for help
[wmv3 @ 0x56333dae9a00] No support for codec wmv3 profile 1.
[wmv3 @ 0x56333dae9a00] Failed setup for format vaapi: hwaccel initialisation returned error.
[h264_vaapi @ 0x56333daeb7c0] No quality level set; using default (20).
[h264_vaapi @ 0x56333daeb7c0] Driver does not support some wanted packed headers (wanted 0xd, found 0).
Output #0, h264, to 'test.h264':
Metadata:
IsVBR : 0
MediaFoundationVersion: 2.112
encoder : Lavf59.34.102
Stream #0:0(eng): Video: h264 (High), vaapi(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 24 fps, 24 tbn
Metadata:
encoder : Lavc59.56.100 h264_vaapi
frame= 1497 fps= 50 q=-0.0 Lsize= 19644kB time=00:01:02.33 bitrate=2581.7kbits/s speed= 2.1x
video:19644kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
然后播放到也正常
root@pc:~# ./ffmpeg/ffprobe test.h264
Input #0, h264, from 'test.h264':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn
root@pc:~# ./ffmpeg/ffplay test.h264
不带-hwaccel vaapi参数
./ffmpeg/ffmpeg -vaapi_device /dev/dri/renderD128 -i wmv.wmv -vf 'format=nv12,hwupload' -c:v h264_vaapi testwmv.mp4
转码也会经过virglrenderer。
root@pc:~# ./ffmpeg/ffmpeg -vaapi_device /dev/dri/renderD128 -i wmv.wmv -vf 'format=nv12,hwupload' -c:v h264_vaapi test.h264
[wmv3 @ 0x562019a69300] Extra data: 8 bits left, value: 0
Input #0, asf, from 'wmv.wmv':
Metadata:
IsVBR : 0
MediaFoundationVersion: 2.112
Duration: 00:01:02.37, start: 0.000000, bitrate: 1032 kb/s
Stream #0:0(eng): Video: wmv3 (Main) (WMV3 / 0x33564D57), yuv420p, 1280x720, 1024 kb/s, SAR 1:1 DAR 16:9, 24 fps, 24 tbr, 1k tbn
File 'test.h264' already exists. Overwrite? [y/N] y
[wmv3 @ 0x562019a7a7c0] Extra data: 8 bits left, value: 0
Stream mapping:
Stream #0:0 -> #0:0 (wmv3 (native) -> h264 (h264_vaapi))
Press [q] to stop, [?] for help
[h264_vaapi @ 0x562019a7c580] No quality level set; using default (20).
[h264_vaapi @ 0x562019a7c580] Driver does not support some wanted packed headers (wanted 0xd, found 0).
Output #0, h264, to 'test.h264':
Metadata:
IsVBR : 0
MediaFoundationVersion: 2.112
encoder : Lavf59.34.102
Stream #0:0(eng): Video: h264 (High), vaapi(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 24 fps, 24 tbn
Metadata:
encoder : Lavc59.56.100 h264_vaapi
frame= 1497 fps= 41 q=-0.0 Lsize= 19644kB time=00:01:02.33 bitrate=2581.7kbits/s speed=1.72x
video:19644kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
查看其编码信息
root@pc:~# ./ffmpeg/ffprobe test.h264
Input #0, h264, from 'test.h264':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn
测试遇到的一些问题:
-
每次改变remote viewer窗口的大小,guest虚机图形界面都会被自动lock,需要重新输入密码登录才可以。文章来源:https://www.toymoban.com/news/detail-628457.html
-
vlc播放视频时,有时会花屏,有时会卡死。文章来源地址https://www.toymoban.com/news/detail-628457.html
到了这里,关于视频编解码(一)之virtio-gpu环境搭建的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!