OpenVINO 2022.3之七：OpenVINO 预处理API提升模型推理性能-Toy模板网

这篇具有很好参考价值的文章主要介绍了OpenVINO 2022.3之七：OpenVINO 预处理API提升模型推理性能。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

OpenVINO 2022.3之七：OpenVINO 预处理API提升模型推理性能

OpenVINO™ 2022.3 提供OpenVINO™ Runtime原生的用于数据预处理的API函数。
如果没有预处理API，那么输入数据的预处理操作只能放在CPU上实现，CPU完成数据预处理后，再将预处理后的数据传给iGPU、VPU等AI加速计算设备进行推理计算。
有了预处理API后，就能将预处理操作集成到在模型执行图中，这样iGPU、VPU 或即将发布的Intel独立显卡都能进行数据预处理，无需依赖CPU，提高了执行效率。

常见数据预处理的操作有：

Mean/Scale Normalization

c++:

// Suppose model's shape is {1, 3, 224, 224}
ppp.input("input").model().set_layout("NCHW"); // N=1, C=3, H=224, W=224
// Mean/Scale has 3 values which matches with C=3
ppp.input("input").preprocess()
  .mean({103.94f, 116.78f, 123.68f}).scale({57.21f, 57.45f, 57.73f});

python:

# Suppose model's shape is {1, 3, 224, 224}
# N=1, C=3, H=224, W=224
ppp.input('input').model().set_layout(Layout('NCHW'))
# Mean/Scale has 3 values which matches with C=3
ppp.input('input').preprocess() \
    .mean([103.94, 116.78, 123.68]).scale([57.21, 57.45, 57.73])

Converting Precision

c++:

// First define data type for your tensor
ppp.input("input").tensor().set_element_type(ov::element::u8);

// Then define preprocessing step
ppp.input("input").preprocess().convert_element_type(ov::element::f32);

// If conversion is needed to `model's` element type, 'f32' can be omitted
ppp.input("input").preprocess().convert_element_type();

python:

# First define data type for your tensor
ppp.input('input').tensor().set_element_type(Type.u8)

# Then define preprocessing step
ppp.input('input').preprocess().convert_element_type(Type.f32)

# If conversion is needed to `model's` element type, 'f32' can be omitted
ppp.input('input').preprocess().convert_element_type()

Converting layout (transposing)

c++:

// First define layout for your tensor
ppp.input("input").tensor().set_layout("NHWC");

// Then define layout of model
ppp.input("input").model().set_layout("NCHW");

std::cout << ppp; // Will print 'implicit layout conversion step'

python:

# First define layout for your tensor
ppp.input('input').tensor().set_layout(Layout('NHWC'))

# Then define layout of model
ppp.input('input').model().set_layout(Layout('NCHW'))

print(ppp)  # Will print 'implicit layout conversion step'

Resizing Image

c++:

ppp.input("input").tensor().set_shape({1, 3, 960, 1280});
ppp.input("input").model().set_layout("??HW");
ppp.input("input").preprocess().resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR, 480, 640);

python:

ppp.input('input').tensor().set_shape([1, 3, 960, 1280])
ppp.input('input').model().set_layout(Layout('??HW'))
ppp.input('input').preprocess()\
    .resize(ResizeAlgorithm.RESIZE_LINEAR, 480, 640)

Color Conversion

c++:

ppp.input("input").tensor().set_color_format(ov::preprocess::ColorFormat::BGR);
ppp.input("input").preprocess().convert_color(ov::preprocess::ColorFormat::RGB);

python:

ppp.input('input').tensor().set_color_format(ColorFormat.BGR)
ppp.input('input').preprocess().convert_color(ColorFormat.RGB)

Color Conversion - NV12/I420

c++:

// This will split original `input` to 2 separate inputs: `input/y' and 'input/uv'
ppp.input("input").tensor().set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES);
ppp.input("input").preprocess().convert_color(ov::preprocess::ColorFormat::RGB);
std::cout << ppp;  // Dump preprocessing steps to see what will happen

python:

# This will split original `input` to 2 separate inputs: `input/y' and 'input/uv'
ppp.input('input').tensor()\
    .set_color_format(ColorFormat.NV12_TWO_PLANES)

ppp.input('input').preprocess()\
    .convert_color(ColorFormat.RGB)
print(ppp)  # Dump preprocessing steps to see what will happen

示例：

改变输入数据的形状：[720, 1280,3] → [1, 3, 640, 640]
改变输入数据的精度：U8 → f32
改变输入数据的颜色通道顺序：BGR → RGB
改变输入数据的布局(layout)：HWC → NCHW
归一化数据：减去均值(mean)，除以标准差(std)

C++代码：

#include <openvino/runtime/core.hpp>
#include <openvino/core/preprocess/pre_post_process.hpp>
#include <openvino/pass/serialize.hpp>


// ========  Step 0: read original model =========
ov::Core core;
std::shared_ptr<ov::Model> model = core.read_model("/path/to/some_model.xml");

// ======== Step 1: Preprocessing ================
ov::preprocess::PrePostProcessor prep(model);
// Declare section of desired application's input format
prep.input().tensor()
       .set_element_type(ov::element::u8)
       .set_layout("NHWC")
       .set_color_format(ov::preprocess::ColorFormat::BGR)
       .set_spatial_dynamic_shape();
// Specify actual model layout
prep.input().model()
       .set_layout("NCHW");
// Explicit preprocessing steps. Layout conversion will be done automatically as last step
prep.input().preprocess()
       .convert_element_type()
       .convert_color(ov::preprocess::ColorFormat::RGB)
       .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
       .mean({123.675f, 116.28f, 103.53f}) // Subtract mean after color conversion
       .scale({58.624f, 57.12f, 57.375f});
// Dump preprocessor
std::cout << "Preprocessor: " << prep << std::endl;
model = prep.build();

// ======== Step 2: Change batch size ================
// In this example we also want to change batch size to increase throughput
ov::set_batch(model, 2);

// ======== Step 3: Save the model ================
std::string xml = "/path/to/some_model_saved.xml";
std::string bin = "/path/to/some_model_saved.bin";
ov::serialize(model, xml, bin);

Python代码：

from openvino.preprocess import PrePostProcessor, ColorFormat, ResizeAlgorithm
from openvino.runtime import Core, Layout, Type, set_batch
# First method - imports
from openvino.runtime import serialize
# Second method - imports
from openvino.runtime.passes import Manager, Serialize

# ========  Step 0: read original model =========
core = Core()
model = core.read_model(model='/path/to/some_model.xml')

# ======== Step 1: Preprocessing ================
ppp = PrePostProcessor(model)
# Declare section of desired application's input format
ppp.input().tensor() \
    .set_element_type(Type.u8) \
    .set_spatial_dynamic_shape() \
    .set_layout(Layout('NHWC')) \
    .set_color_format(ColorFormat.BGR)

# Specify actual model layout
ppp.input().model().set_layout(Layout('NCHW'))

# Explicit preprocessing steps. Layout conversion will be done automatically as last step
ppp.input().preprocess() \
    .convert_element_type() \
    .convert_color(ColorFormat.RGB) \
    .resize(ResizeAlgorithm.RESIZE_LINEAR) \
    .mean([123.675, 116.28, 103.53]) \
    .scale([58.624, 57.12, 57.375])

# Dump preprocessor
print(f'Dump preprocessor: {ppp}')
model = ppp.build()

# ======== Step 2: Change batch size ================
# In this example we also want to change batch size to increase throughput
set_batch(model, 2)

# ======== Step 3: Save the model ================
# First method - using serialize runtime wrapper
serialize(model, '/path/to/some_model_saved.xml', '/path/to/some_model_saved.bin')

# Second method - using Manager and Serialize pass
manager = Manager()
manager.register_pass(Serialize('/path/to/some_model_saved.xml', '/path/to/some_model_saved.bin'))
manager.run_passes(model)

实战：

Opencv代码：

std::vector<float> paddings(3);       //scale, half_h, half_w
cv::Mat resized_img = letterbox(img, paddings); //resize to (640,640) by letterbox
// BGR->RGB, u8(0-255)->f32(0.0-1.0), HWC->NCHW
cv::Mat blob = cv::dnn::blobFromImage(resized_img, 1 / 255.0, cv::Size(640, 640), cv::Scalar(0, 0, 0), true);

OpenVINO Runtime API代码：

	// Step 3. Inizialize Preprocessing for the model
	ov::preprocess::PrePostProcessor ppp = ov::preprocess::PrePostProcessor(model);
	// Specify input image format
	ppp.input().tensor().set_element_type(ov::element::u8).set_layout("NHWC").set_color_format(ov::preprocess::ColorFormat::BGR);
	// Specify preprocess pipeline to input image without resizing
	ppp.input().preprocess().convert_element_type(ov::element::f32).convert_color(ov::preprocess::ColorFormat::RGB).scale({ 255., 255., 255. });
	//  Specify model's input layout
	ppp.input().model().set_layout("NCHW");
	// Specify output results format
	ppp.output().tensor().set_element_type(ov::element::f32);
	// Embed above steps in the graph
	model = ppp.build();
	compiled_model = core.compile_model(model, "CPU");

速度对比能加速 10~20% 左右文章来源地址https://www.toymoban.com/news/detail-439467.html

到了这里，关于OpenVINO 2022.3之七：OpenVINO 预处理API提升模型推理性能的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！