1160 lines
37 KiB
ReStructuredText
1160 lines
37 KiB
ReStructuredText
Convert and Optimize YOLOv10 with OpenVINO
|
||
==========================================
|
||
|
||
Real-time object detection aims to accurately predict object categories
|
||
and positions in images with low latency. The YOLO series has been at
|
||
the forefront of this research due to its balance between performance
|
||
and efficiency. However, reliance on NMS and architectural
|
||
inefficiencies have hindered optimal performance. YOLOv10 addresses
|
||
these issues by introducing consistent dual assignments for NMS-free
|
||
training and a holistic efficiency-accuracy driven model design
|
||
strategy.
|
||
|
||
YOLOv10, built on the `Ultralytics Python
|
||
package <https://pypi.org/project/ultralytics/>`__ by researchers at
|
||
`Tsinghua University <https://www.tsinghua.edu.cn/en/>`__, introduces a
|
||
new approach to real-time object detection, addressing both the
|
||
post-processing and model architecture deficiencies found in previous
|
||
YOLO versions. By eliminating non-maximum suppression (NMS) and
|
||
optimizing various model components, YOLOv10 achieves state-of-the-art
|
||
performance with significantly reduced computational overhead. Extensive
|
||
experiments demonstrate its superior accuracy-latency trade-offs across
|
||
multiple model scales.
|
||
|
||
.. figure:: https://github.com/ultralytics/ultralytics/assets/26833433/f9b1bec0-928e-41ce-a205-e12db3c4929a
|
||
:alt: yolov10-approach.png
|
||
|
||
yolov10-approach.png
|
||
|
||
More details about model architecture you can find in original
|
||
`repo <https://github.com/THU-MIG/yolov10>`__,
|
||
`paper <https://arxiv.org/abs/2405.14458>`__ and `Ultralytics
|
||
documentation <https://docs.ultralytics.com/models/yolov10/>`__.
|
||
|
||
This tutorial demonstrates step-by-step instructions on how to run and
|
||
optimize PyTorch YOLO V10 with OpenVINO.
|
||
|
||
The tutorial consists of the following steps:
|
||
|
||
- Prepare PyTorch model
|
||
- Convert PyTorch model to OpenVINO IR
|
||
- Run model inference with OpenVINO
|
||
- Prepare and run optimization pipeline using NNCF
|
||
- Compare performance of the FP16 and quantized models.
|
||
- Run optimized model inference on video
|
||
- Launch interactive Gradio demo
|
||
|
||
Table of contents:
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
- `Prerequisites <#prerequisites>`__
|
||
- `Download PyTorch model <#download-pytorch-model>`__
|
||
- `Export PyTorch model to OpenVINO IR
|
||
Format <#export-pytorch-model-to-openvino-ir-format>`__
|
||
- `Run OpenVINO Inference on AUTO device using Ultralytics
|
||
API <#run-openvino-inference-on-auto-device-using-ultralytics-api>`__
|
||
- `Run OpenVINO Inference on selected device using Ultralytics
|
||
API <#run-openvino-inference-on-selected-device-using-ultralytics-api>`__
|
||
- `Optimize model using NNCF Post-training Quantization
|
||
API <#optimize-model-using-nncf-post-training-quantization-api>`__
|
||
|
||
- `Prepare Quantization Dataset <#prepare-quantization-dataset>`__
|
||
- `Quantize and Save INT8 model <#quantize-and-save-int8-model>`__
|
||
|
||
- `Run Optimized Model Inference <#run-optimized-model-inference>`__
|
||
|
||
- `Run Optimized Model on AUTO
|
||
device <#run-optimized-model-on-auto-device>`__
|
||
- `Run Optimized Model Inference on selected
|
||
device <#run-optimized-model-inference-on-selected-device>`__
|
||
|
||
- `Compare the Original and Quantized
|
||
Models <#compare-the-original-and-quantized-models>`__
|
||
|
||
- `Model size <#model-size>`__
|
||
- `Performance <#performance>`__
|
||
- `FP16 model performance <#fp16-model-performance>`__
|
||
- `Int8 model performance <#int8-model-performance>`__
|
||
|
||
- `Live demo <#live-demo>`__
|
||
|
||
- `Gradio Interactive Demo <#gradio-interactive-demo>`__
|
||
|
||
Prerequisites
|
||
-------------
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
import os
|
||
|
||
os.environ["GIT_CLONE_PROTECTION_ACTIVE"] = "false"
|
||
|
||
%pip install -Uq pip
|
||
%pip install -q "git+https://github.com/openvinotoolkit/nncf.git"
|
||
%pip install --pre -Uq openvino --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
|
||
%pip install -q "git+https://github.com/THU-MIG/yolov10.git" --extra-index-url https://download.pytorch.org/whl/cpu
|
||
%pip install -q "torch>=2.1" "torchvision>=0.16" tqdm opencv-python "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
WARNING: Skipping openvino as it is not installed.
|
||
WARNING: Skipping openvino-dev as it is not installed.
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
from pathlib import Path
|
||
|
||
# Fetch `notebook_utils` module
|
||
import requests
|
||
|
||
r = requests.get(
|
||
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
|
||
)
|
||
|
||
open("notebook_utils.py", "w").write(r.text)
|
||
|
||
from notebook_utils import download_file, VideoPlayer
|
||
|
||
Download PyTorch model
|
||
----------------------
|
||
|
||
|
||
|
||
There are several version of `YOLO
|
||
V10 <https://github.com/THU-MIG/yolov10/tree/main?tab=readme-ov-file#performance>`__
|
||
models provided by model authors. Each of them has different
|
||
characteristics depends on number of training parameters, performance
|
||
and accuracy. For demonstration purposes we will use ``yolov10n``, but
|
||
the same steps are also applicable to other models in YOLO V10 series.
|
||
|
||
.. code:: ipython3
|
||
|
||
models_dir = Path("./models")
|
||
models_dir.mkdir(exist_ok=True)
|
||
|
||
.. code:: ipython3
|
||
|
||
model_weights_url = "https://github.com/jameslahm/yolov10/releases/download/v1.0/yolov10n.pt"
|
||
file_name = model_weights_url.split("/")[-1]
|
||
model_name = file_name.replace(".pt", "")
|
||
|
||
download_file(model_weights_url, directory=models_dir)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
'models/yolov10n.pt' already exists.
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
PosixPath('/home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/models/yolov10n.pt')
|
||
|
||
|
||
|
||
Export PyTorch model to OpenVINO IR Format
|
||
------------------------------------------
|
||
|
||
|
||
|
||
As it was discussed before, YOLO V10 code is designed on top of
|
||
`Ultralytics <https://docs.ultralytics.com/>`__ library and has similar
|
||
interface with YOLO V8 (You can check `YOLO V8
|
||
notebooks <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov8-optimization>`__
|
||
for more detailed instruction how to work with Ultralytics API).
|
||
Ultralytics support OpenVINO model export using
|
||
`export <https://docs.ultralytics.com/modes/export/>`__ method of model
|
||
class. Additionally, we can specify parameters responsible for target
|
||
input size, static or dynamic input shapes and model precision
|
||
(FP32/FP16/INT8). INT8 quantization can be additionally performed on
|
||
export stage, but for making approach more flexible, we consider how to
|
||
perform quantization using
|
||
`NNCF <https://github.com/openvinotoolkit/nncf>`__.
|
||
|
||
.. code:: ipython3
|
||
|
||
import types
|
||
from ultralytics.utils import ops, yaml_load, yaml_save
|
||
from ultralytics import YOLOv10
|
||
import torch
|
||
|
||
detection_labels = {
|
||
0: "person",
|
||
1: "bicycle",
|
||
2: "car",
|
||
3: "motorcycle",
|
||
4: "airplane",
|
||
5: "bus",
|
||
6: "train",
|
||
7: "truck",
|
||
8: "boat",
|
||
9: "traffic light",
|
||
10: "fire hydrant",
|
||
11: "stop sign",
|
||
12: "parking meter",
|
||
13: "bench",
|
||
14: "bird",
|
||
15: "cat",
|
||
16: "dog",
|
||
17: "horse",
|
||
18: "sheep",
|
||
19: "cow",
|
||
20: "elephant",
|
||
21: "bear",
|
||
22: "zebra",
|
||
23: "giraffe",
|
||
24: "backpack",
|
||
25: "umbrella",
|
||
26: "handbag",
|
||
27: "tie",
|
||
28: "suitcase",
|
||
29: "frisbee",
|
||
30: "skis",
|
||
31: "snowboard",
|
||
32: "sports ball",
|
||
33: "kite",
|
||
34: "baseball bat",
|
||
35: "baseball glove",
|
||
36: "skateboard",
|
||
37: "surfboard",
|
||
38: "tennis racket",
|
||
39: "bottle",
|
||
40: "wine glass",
|
||
41: "cup",
|
||
42: "fork",
|
||
43: "knife",
|
||
44: "spoon",
|
||
45: "bowl",
|
||
46: "banana",
|
||
47: "apple",
|
||
48: "sandwich",
|
||
49: "orange",
|
||
50: "broccoli",
|
||
51: "carrot",
|
||
52: "hot dog",
|
||
53: "pizza",
|
||
54: "donut",
|
||
55: "cake",
|
||
56: "chair",
|
||
57: "couch",
|
||
58: "potted plant",
|
||
59: "bed",
|
||
60: "dining table",
|
||
61: "toilet",
|
||
62: "tv",
|
||
63: "laptop",
|
||
64: "mouse",
|
||
65: "remote",
|
||
66: "keyboard",
|
||
67: "cell phone",
|
||
68: "microwave",
|
||
69: "oven",
|
||
70: "toaster",
|
||
71: "sink",
|
||
72: "refrigerator",
|
||
73: "book",
|
||
74: "clock",
|
||
75: "vase",
|
||
76: "scissors",
|
||
77: "teddy bear",
|
||
78: "hair drier",
|
||
79: "toothbrush",
|
||
}
|
||
|
||
|
||
def v10_det_head_forward(self, x):
|
||
one2one = self.forward_feat([xi.detach() for xi in x], self.one2one_cv2, self.one2one_cv3)
|
||
if not self.export:
|
||
one2many = super().forward(x)
|
||
|
||
if not self.training:
|
||
one2one = self.inference(one2one)
|
||
if not self.export:
|
||
return {"one2many": one2many, "one2one": one2one}
|
||
else:
|
||
assert self.max_det != -1
|
||
boxes, scores, labels = ops.v10postprocess(one2one.permute(0, 2, 1), self.max_det, self.nc)
|
||
return torch.cat(
|
||
[boxes, scores.unsqueeze(-1), labels.unsqueeze(-1).to(boxes.dtype)],
|
||
dim=-1,
|
||
)
|
||
else:
|
||
return {"one2many": one2many, "one2one": one2one}
|
||
|
||
|
||
ov_model_path = models_dir / f"{model_name}_openvino_model/{model_name}.xml"
|
||
if not ov_model_path.exists():
|
||
model = YOLOv10(models_dir / file_name)
|
||
model.model.model[-1].forward = types.MethodType(v10_det_head_forward, model.model.model[-1])
|
||
model.export(format="openvino", dynamic=True, half=True)
|
||
config = yaml_load(ov_model_path.parent / "metadata.yaml")
|
||
config["names"] = detection_labels
|
||
yaml_save(ov_model_path.parent / "metadata.yaml", config)
|
||
|
||
Run OpenVINO Inference on AUTO device using Ultralytics API
|
||
-----------------------------------------------------------
|
||
|
||
|
||
|
||
Now, when we exported model to OpenVINO, we can load it directly into
|
||
YOLOv10 class, where automatic inference backend will provide
|
||
easy-to-use user experience to run OpenVINO YOLOv10 model on the similar
|
||
level like for original PyTorch model. The code bellow demonstrates how
|
||
to run inference OpenVINO exported model with Ultralytics API on single
|
||
image. `AUTO
|
||
device <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/auto-device>`__
|
||
will be used for launching model.
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_yolo_model = YOLOv10(ov_model_path.parent, task="detect")
|
||
|
||
.. code:: ipython3
|
||
|
||
from PIL import Image
|
||
|
||
IMAGE_PATH = Path("./data/coco_bike.jpg")
|
||
download_file(
|
||
url="https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg",
|
||
filename=IMAGE_PATH.name,
|
||
directory=IMAGE_PATH.parent,
|
||
)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
'data/coco_bike.jpg' already exists.
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
PosixPath('/home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/data/coco_bike.jpg')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
res = ov_yolo_model(IMAGE_PATH, iou=0.45, conf=0.2)
|
||
Image.fromarray(res[0].plot()[:, :, ::-1])
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Loading models/yolov10n_openvino_model for OpenVINO inference...
|
||
requirements: Ultralytics requirement ['openvino>=2024.0.0'] not found, attempting AutoUpdate...
|
||
requirements: ❌ AutoUpdate skipped (offline)
|
||
Using OpenVINO LATENCY mode for batch=1 inference...
|
||
|
||
image 1/1 /home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/data/coco_bike.jpg: 640x640 1 bicycle, 2 cars, 1 motorcycle, 1 dog, 72.0ms
|
||
Speed: 25.6ms preprocess, 72.0ms inference, 0.6ms postprocess per image at shape (1, 3, 640, 640)
|
||
|
||
|
||
|
||
|
||
.. image:: yolov10-optimization-with-output_files/yolov10-optimization-with-output_13_1.png
|
||
|
||
|
||
|
||
Run OpenVINO Inference on selected device using Ultralytics API
|
||
---------------------------------------------------------------
|
||
|
||
|
||
|
||
In this part of notebook you can select inference device for running
|
||
model inference to compare results with AUTO.
|
||
|
||
.. code:: ipython3
|
||
|
||
import openvino as ov
|
||
|
||
import ipywidgets as widgets
|
||
|
||
core = ov.Core()
|
||
|
||
device = widgets.Dropdown(
|
||
options=core.available_devices + ["AUTO"],
|
||
value="CPU",
|
||
description="Device:",
|
||
disabled=False,
|
||
)
|
||
|
||
device
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_model = core.read_model(ov_model_path)
|
||
|
||
# load model on selected device
|
||
if "GPU" in device.value or "NPU" in device.value:
|
||
ov_model.reshape({0: [1, 3, 640, 640]})
|
||
ov_config = {}
|
||
if "GPU" in device.value:
|
||
ov_config = {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"}
|
||
det_compiled_model = core.compile_model(ov_model, device.value, ov_config)
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_yolo_model.predictor.model.ov_compiled_model = det_compiled_model
|
||
|
||
.. code:: ipython3
|
||
|
||
res = ov_yolo_model(IMAGE_PATH, iou=0.45, conf=0.2)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
|
||
image 1/1 /home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/data/coco_bike.jpg: 640x640 1 bicycle, 2 cars, 1 motorcycle, 1 dog, 29.1ms
|
||
Speed: 3.2ms preprocess, 29.1ms inference, 0.3ms postprocess per image at shape (1, 3, 640, 640)
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
Image.fromarray(res[0].plot()[:, :, ::-1])
|
||
|
||
|
||
|
||
|
||
.. image:: yolov10-optimization-with-output_files/yolov10-optimization-with-output_19_0.png
|
||
|
||
|
||
|
||
Optimize model using NNCF Post-training Quantization API
|
||
--------------------------------------------------------
|
||
|
||
|
||
|
||
`NNCF <https://github.com/openvinotoolkit/nncf>`__ provides a suite of
|
||
advanced algorithms for Neural Networks inference optimization in
|
||
OpenVINO with minimal accuracy drop. We will use 8-bit quantization in
|
||
post-training mode (without the fine-tuning pipeline) to optimize
|
||
YOLOv10.
|
||
|
||
The optimization process contains the following steps:
|
||
|
||
1. Create a Dataset for quantization.
|
||
2. Run ``nncf.quantize`` for getting an optimized model.
|
||
3. Serialize OpenVINO IR model, using the ``openvino.save_model``
|
||
function.
|
||
|
||
Quantization is time and memory consuming process, you can skip this
|
||
step using checkbox bellow:
|
||
|
||
.. code:: ipython3
|
||
|
||
import ipywidgets as widgets
|
||
|
||
int8_model_det_path = models_dir / "int8" / f"{model_name}_openvino_model/{model_name}.xml"
|
||
ov_yolo_int8_model = None
|
||
|
||
to_quantize = widgets.Checkbox(
|
||
value=True,
|
||
description="Quantization",
|
||
disabled=False,
|
||
)
|
||
|
||
to_quantize
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Checkbox(value=True, description='Quantization')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
# Fetch skip_kernel_extension module
|
||
r = requests.get(
|
||
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
|
||
)
|
||
open("skip_kernel_extension.py", "w").write(r.text)
|
||
|
||
%load_ext skip_kernel_extension
|
||
|
||
Prepare Quantization Dataset
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
For starting quantization, we need to prepare dataset. We will use
|
||
validation subset from `MS COCO dataset <https://cocodataset.org/>`__
|
||
for model quantization and Ultralytics validation data loader for
|
||
preparing input data.
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
from zipfile import ZipFile
|
||
|
||
from ultralytics.data.utils import DATASETS_DIR
|
||
|
||
if not int8_model_det_path.exists():
|
||
|
||
DATA_URL = "http://images.cocodataset.org/zips/val2017.zip"
|
||
LABELS_URL = "https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels-segments.zip"
|
||
CFG_URL = "https://raw.githubusercontent.com/ultralytics/ultralytics/v8.1.0/ultralytics/cfg/datasets/coco.yaml"
|
||
|
||
OUT_DIR = DATASETS_DIR
|
||
|
||
DATA_PATH = OUT_DIR / "val2017.zip"
|
||
LABELS_PATH = OUT_DIR / "coco2017labels-segments.zip"
|
||
CFG_PATH = OUT_DIR / "coco.yaml"
|
||
|
||
download_file(DATA_URL, DATA_PATH.name, DATA_PATH.parent)
|
||
download_file(LABELS_URL, LABELS_PATH.name, LABELS_PATH.parent)
|
||
download_file(CFG_URL, CFG_PATH.name, CFG_PATH.parent)
|
||
|
||
if not (OUT_DIR / "coco/labels").exists():
|
||
with ZipFile(LABELS_PATH, "r") as zip_ref:
|
||
zip_ref.extractall(OUT_DIR)
|
||
with ZipFile(DATA_PATH, "r") as zip_ref:
|
||
zip_ref.extractall(OUT_DIR / "coco/images")
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
from ultralytics.utils import DEFAULT_CFG
|
||
from ultralytics.cfg import get_cfg
|
||
from ultralytics.data.converter import coco80_to_coco91_class
|
||
from ultralytics.data.utils import check_det_dataset
|
||
|
||
if not int8_model_det_path.exists():
|
||
args = get_cfg(cfg=DEFAULT_CFG)
|
||
args.data = str(CFG_PATH)
|
||
det_validator = ov_yolo_model.task_map[ov_yolo_model.task]["validator"](args=args)
|
||
|
||
det_validator.data = check_det_dataset(args.data)
|
||
det_validator.stride = 32
|
||
det_data_loader = det_validator.get_dataloader(OUT_DIR / "coco", 1)
|
||
|
||
NNCF provides ``nncf.Dataset`` wrapper for using native framework
|
||
dataloaders in quantization pipeline. Additionally, we specify transform
|
||
function that will be responsible for preparing input data in model
|
||
expected format.
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
import nncf
|
||
from typing import Dict
|
||
|
||
|
||
def transform_fn(data_item:Dict):
|
||
"""
|
||
Quantization transform function. Extracts and preprocess input data from dataloader item for quantization.
|
||
Parameters:
|
||
data_item: Dict with data item produced by DataLoader during iteration
|
||
Returns:
|
||
input_tensor: Input data for quantization
|
||
"""
|
||
input_tensor = det_validator.preprocess(data_item)['img'].numpy()
|
||
return input_tensor
|
||
|
||
if not int8_model_det_path.exists():
|
||
quantization_dataset = nncf.Dataset(det_data_loader, transform_fn)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, openvino
|
||
|
||
|
||
Quantize and Save INT8 model
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
The ``nncf.quantize`` function provides an interface for model
|
||
quantization. It requires an instance of the OpenVINO Model and
|
||
quantization dataset. Optionally, some additional parameters for the
|
||
configuration quantization process (number of samples for quantization,
|
||
preset, ignored scope, etc.) can be provided. YOLOv10 model contains
|
||
non-ReLU activation functions, which require asymmetric quantization of
|
||
activations. To achieve a better result, we will use a ``mixed``
|
||
quantization preset. It provides symmetric quantization of weights and
|
||
asymmetric quantization of activations.
|
||
|
||
**Note**: Model post-training quantization is time-consuming process.
|
||
Be patient, it can take several minutes depending on your hardware.
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
import shutil
|
||
|
||
if not int8_model_det_path.exists():
|
||
quantized_det_model = nncf.quantize(
|
||
ov_model,
|
||
quantization_dataset,
|
||
preset=nncf.QuantizationPreset.MIXED,
|
||
)
|
||
|
||
ov.save_model(quantized_det_model, int8_model_det_path)
|
||
shutil.copy(ov_model_path.parent / "metadata.yaml", int8_model_det_path.parent / "metadata.yaml")
|
||
|
||
Run Optimized Model Inference
|
||
-----------------------------
|
||
|
||
|
||
|
||
The way of usage INT8 quantized model is the same like for model before
|
||
quantization. Let’s check inference result of quantized model on single
|
||
image
|
||
|
||
Run Optimized Model on AUTO device
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
ov_yolo_int8_model = YOLOv10(int8_model_det_path.parent, task="detect")
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
res = ov_yolo_int8_model(IMAGE_PATH, iou=0.45, conf=0.2)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Loading models/int8/yolov10n_openvino_model for OpenVINO inference...
|
||
requirements: Ultralytics requirement ['openvino>=2024.0.0'] not found, attempting AutoUpdate...
|
||
requirements: ❌ AutoUpdate skipped (offline)
|
||
Using OpenVINO LATENCY mode for batch=1 inference...
|
||
|
||
image 1/1 /home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/data/coco_bike.jpg: 640x640 1 bicycle, 3 cars, 2 motorcycles, 1 dog, 92.3ms
|
||
Speed: 3.7ms preprocess, 92.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 640)
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
Image.fromarray(res[0].plot()[:, :, ::-1])
|
||
|
||
|
||
|
||
|
||
.. image:: yolov10-optimization-with-output_files/yolov10-optimization-with-output_34_0.png
|
||
|
||
|
||
|
||
Run Optimized Model Inference on selected device
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
device
|
||
|
||
.. code:: ipython3
|
||
|
||
%%skip not $to_quantize.value
|
||
|
||
ov_config = {}
|
||
if "GPU" in device.value or "NPU" in device.value:
|
||
ov_model.reshape({0: [1, 3, 640, 640]})
|
||
ov_config = {}
|
||
if "GPU" in device.value:
|
||
ov_config = {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"}
|
||
|
||
quantized_det_model = core.read_model(int8_model_det_path)
|
||
quantized_det_compiled_model = core.compile_model(quantized_det_model, device.value, ov_config)
|
||
|
||
ov_yolo_int8_model.predictor.model.ov_compiled_model = quantized_det_compiled_model
|
||
|
||
res = ov_yolo_int8_model(IMAGE_PATH, iou=0.45, conf=0.2)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
|
||
image 1/1 /home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/yolov10-optimization/data/coco_bike.jpg: 640x640 1 bicycle, 3 cars, 2 motorcycles, 1 dog, 26.5ms
|
||
Speed: 7.4ms preprocess, 26.5ms inference, 0.3ms postprocess per image at shape (1, 3, 640, 640)
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
Image.fromarray(res[0].plot()[:, :, ::-1])
|
||
|
||
|
||
|
||
|
||
.. image:: yolov10-optimization-with-output_files/yolov10-optimization-with-output_38_0.png
|
||
|
||
|
||
|
||
Compare the Original and Quantized Models
|
||
-----------------------------------------
|
||
|
||
|
||
|
||
Model size
|
||
~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_model_weights = ov_model_path.with_suffix(".bin")
|
||
print(f"Size of FP16 model is {ov_model_weights.stat().st_size / 1024 / 1024:.2f} MB")
|
||
if int8_model_det_path.exists():
|
||
ov_int8_weights = int8_model_det_path.with_suffix(".bin")
|
||
print(f"Size of model with INT8 compressed weights is {ov_int8_weights.stat().st_size / 1024 / 1024:.2f} MB")
|
||
print(f"Compression rate for INT8 model: {ov_model_weights.stat().st_size / ov_int8_weights.stat().st_size:.3f}")
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Size of FP16 model is 4.39 MB
|
||
Size of model with INT8 compressed weights is 2.25 MB
|
||
Compression rate for INT8 model: 1.954
|
||
|
||
|
||
Performance
|
||
~~~~~~~~~~~
|
||
|
||
|
||
|
||
FP16 model performance
|
||
~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
!benchmark_app -m $ov_model_path -d $device.value -api async -shape "[1,3,640,640]" -t 15
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
[Step 1/11] Parsing and validating input arguments
|
||
[ INFO ] Parsing input parameters
|
||
[Step 2/11] Loading OpenVINO Runtime
|
||
[ INFO ] OpenVINO:
|
||
[ INFO ] Build ................................. 2024.2.0-15496-17f8e86e5f2-releases/2024/2
|
||
[ INFO ]
|
||
[ INFO ] Device info:
|
||
[ INFO ] CPU
|
||
[ INFO ] Build ................................. 2024.2.0-15496-17f8e86e5f2-releases/2024/2
|
||
[ INFO ]
|
||
[ INFO ]
|
||
[Step 3/11] Setting device configuration
|
||
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
|
||
[Step 4/11] Reading model files
|
||
[ INFO ] Loading model files
|
||
[ INFO ] Read model took 31.92 ms
|
||
[ INFO ] Original model I/O parameters:
|
||
[ INFO ] Model inputs:
|
||
[ INFO ] x (node: x) : f32 / [...] / [?,3,?,?]
|
||
[ INFO ] Model outputs:
|
||
[ INFO ] ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [?,300,6]
|
||
[Step 5/11] Resizing model to match image sizes and given batch
|
||
[ INFO ] Model batch size: 1
|
||
[ INFO ] Reshaping model: 'x': [1,3,640,640]
|
||
[ INFO ] Reshape model took 17.77 ms
|
||
[Step 6/11] Configuring input of the model
|
||
[ INFO ] Model inputs:
|
||
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,640,640]
|
||
[ INFO ] Model outputs:
|
||
[ INFO ] ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [1,300,6]
|
||
[Step 7/11] Loading the model to the device
|
||
[ INFO ] Compile model took 303.83 ms
|
||
[Step 8/11] Querying optimal runtime parameters
|
||
[ INFO ] Model:
|
||
[ INFO ] NETWORK_NAME: Model0
|
||
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12
|
||
[ INFO ] NUM_STREAMS: 12
|
||
[ INFO ] INFERENCE_NUM_THREADS: 36
|
||
[ INFO ] PERF_COUNT: NO
|
||
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
|
||
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
|
||
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
|
||
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
|
||
[ INFO ] ENABLE_CPU_PINNING: True
|
||
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
|
||
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
|
||
[ INFO ] ENABLE_HYPER_THREADING: True
|
||
[ INFO ] EXECUTION_DEVICES: ['CPU']
|
||
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
|
||
[ INFO ] LOG_LEVEL: Level.NO
|
||
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
|
||
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
|
||
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
|
||
[ INFO ] AFFINITY: Affinity.CORE
|
||
[Step 9/11] Creating infer requests and preparing input tensors
|
||
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
|
||
[ INFO ] Fill input 'x' with random values
|
||
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration)
|
||
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
|
||
[ INFO ] First inference took 30.60 ms
|
||
[Step 11/11] Dumping statistics report
|
||
[ INFO ] Execution Devices:['CPU']
|
||
[ INFO ] Count: 2424 iterations
|
||
[ INFO ] Duration: 15093.22 ms
|
||
[ INFO ] Latency:
|
||
[ INFO ] Median: 72.34 ms
|
||
[ INFO ] Average: 74.46 ms
|
||
[ INFO ] Min: 45.87 ms
|
||
[ INFO ] Max: 147.25 ms
|
||
[ INFO ] Throughput: 160.60 FPS
|
||
|
||
|
||
Int8 model performance
|
||
~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
if int8_model_det_path.exists():
|
||
!benchmark_app -m $int8_model_det_path -d $device.value -api async -shape "[1,3,640,640]" -t 15
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
[Step 1/11] Parsing and validating input arguments
|
||
[ INFO ] Parsing input parameters
|
||
[Step 2/11] Loading OpenVINO Runtime
|
||
[ INFO ] OpenVINO:
|
||
[ INFO ] Build ................................. 2024.2.0-15496-17f8e86e5f2-releases/2024/2
|
||
[ INFO ]
|
||
[ INFO ] Device info:
|
||
[ INFO ] CPU
|
||
[ INFO ] Build ................................. 2024.2.0-15496-17f8e86e5f2-releases/2024/2
|
||
[ INFO ]
|
||
[ INFO ]
|
||
[Step 3/11] Setting device configuration
|
||
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
|
||
[Step 4/11] Reading model files
|
||
[ INFO ] Loading model files
|
||
[ INFO ] Read model took 38.75 ms
|
||
[ INFO ] Original model I/O parameters:
|
||
[ INFO ] Model inputs:
|
||
[ INFO ] x (node: x) : f32 / [...] / [?,3,?,?]
|
||
[ INFO ] Model outputs:
|
||
[ INFO ] ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [?,300,6]
|
||
[Step 5/11] Resizing model to match image sizes and given batch
|
||
[ INFO ] Model batch size: 1
|
||
[ INFO ] Reshaping model: 'x': [1,3,640,640]
|
||
[ INFO ] Reshape model took 18.33 ms
|
||
[Step 6/11] Configuring input of the model
|
||
[ INFO ] Model inputs:
|
||
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,640,640]
|
||
[ INFO ] Model outputs:
|
||
[ INFO ] ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [1,300,6]
|
||
[Step 7/11] Loading the model to the device
|
||
[ INFO ] Compile model took 622.99 ms
|
||
[Step 8/11] Querying optimal runtime parameters
|
||
[ INFO ] Model:
|
||
[ INFO ] NETWORK_NAME: Model0
|
||
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 18
|
||
[ INFO ] NUM_STREAMS: 18
|
||
[ INFO ] INFERENCE_NUM_THREADS: 36
|
||
[ INFO ] PERF_COUNT: NO
|
||
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
|
||
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
|
||
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
|
||
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
|
||
[ INFO ] ENABLE_CPU_PINNING: True
|
||
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
|
||
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
|
||
[ INFO ] ENABLE_HYPER_THREADING: True
|
||
[ INFO ] EXECUTION_DEVICES: ['CPU']
|
||
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
|
||
[ INFO ] LOG_LEVEL: Level.NO
|
||
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
|
||
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
|
||
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
|
||
[ INFO ] AFFINITY: Affinity.CORE
|
||
[Step 9/11] Creating infer requests and preparing input tensors
|
||
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
|
||
[ INFO ] Fill input 'x' with random values
|
||
[Step 10/11] Measuring performance (Start inference asynchronously, 18 inference requests, limits: 15000 ms duration)
|
||
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
|
||
[ INFO ] First inference took 28.26 ms
|
||
[Step 11/11] Dumping statistics report
|
||
[ INFO ] Execution Devices:['CPU']
|
||
[ INFO ] Count: 5886 iterations
|
||
[ INFO ] Duration: 15067.10 ms
|
||
[ INFO ] Latency:
|
||
[ INFO ] Median: 44.39 ms
|
||
[ INFO ] Average: 45.89 ms
|
||
[ INFO ] Min: 29.73 ms
|
||
[ INFO ] Max: 110.52 ms
|
||
[ INFO ] Throughput: 390.65 FPS
|
||
|
||
|
||
Live demo
|
||
---------
|
||
|
||
|
||
|
||
The following code runs model inference on a video:
|
||
|
||
.. code:: ipython3
|
||
|
||
import collections
|
||
import time
|
||
from IPython import display
|
||
import cv2
|
||
import numpy as np
|
||
|
||
|
||
# Main processing function to run object detection.
|
||
def run_object_detection(
|
||
source=0,
|
||
flip=False,
|
||
use_popup=False,
|
||
skip_first_frames=0,
|
||
det_model=ov_yolo_int8_model,
|
||
device=device.value,
|
||
):
|
||
player = None
|
||
try:
|
||
# Create a video player to play with target fps.
|
||
player = VideoPlayer(source=source, flip=flip, fps=30, skip_first_frames=skip_first_frames)
|
||
# Start capturing.
|
||
player.start()
|
||
if use_popup:
|
||
title = "Press ESC to Exit"
|
||
cv2.namedWindow(winname=title, flags=cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)
|
||
|
||
processing_times = collections.deque()
|
||
while True:
|
||
# Grab the frame.
|
||
frame = player.next()
|
||
if frame is None:
|
||
print("Source ended")
|
||
break
|
||
# If the frame is larger than full HD, reduce size to improve the performance.
|
||
scale = 1280 / max(frame.shape)
|
||
if scale < 1:
|
||
frame = cv2.resize(
|
||
src=frame,
|
||
dsize=None,
|
||
fx=scale,
|
||
fy=scale,
|
||
interpolation=cv2.INTER_AREA,
|
||
)
|
||
# Get the results.
|
||
input_image = np.array(frame)
|
||
|
||
start_time = time.time()
|
||
detections = det_model(input_image, iou=0.45, conf=0.2, verbose=False)
|
||
stop_time = time.time()
|
||
frame = detections[0].plot()
|
||
|
||
processing_times.append(stop_time - start_time)
|
||
# Use processing times from last 200 frames.
|
||
if len(processing_times) > 200:
|
||
processing_times.popleft()
|
||
|
||
_, f_width = frame.shape[:2]
|
||
# Mean processing time [ms].
|
||
processing_time = np.mean(processing_times) * 1000
|
||
fps = 1000 / processing_time
|
||
cv2.putText(
|
||
img=frame,
|
||
text=f"Inference time: {processing_time:.1f}ms ({fps:.1f} FPS)",
|
||
org=(20, 40),
|
||
fontFace=cv2.FONT_HERSHEY_COMPLEX,
|
||
fontScale=f_width / 1000,
|
||
color=(0, 0, 255),
|
||
thickness=1,
|
||
lineType=cv2.LINE_AA,
|
||
)
|
||
# Use this workaround if there is flickering.
|
||
if use_popup:
|
||
cv2.imshow(winname=title, mat=frame)
|
||
key = cv2.waitKey(1)
|
||
# escape = 27
|
||
if key == 27:
|
||
break
|
||
else:
|
||
# Encode numpy array to jpg.
|
||
_, encoded_img = cv2.imencode(ext=".jpg", img=frame, params=[cv2.IMWRITE_JPEG_QUALITY, 100])
|
||
# Create an IPython image.
|
||
i = display.Image(data=encoded_img)
|
||
# Display the image in this notebook.
|
||
display.clear_output(wait=True)
|
||
display.display(i)
|
||
# ctrl-c
|
||
except KeyboardInterrupt:
|
||
print("Interrupted")
|
||
# any different error
|
||
except RuntimeError as e:
|
||
print(e)
|
||
finally:
|
||
if player is not None:
|
||
# Stop capturing.
|
||
player.stop()
|
||
if use_popup:
|
||
cv2.destroyAllWindows()
|
||
|
||
.. code:: ipython3
|
||
|
||
use_int8 = widgets.Checkbox(
|
||
value=ov_yolo_int8_model is not None,
|
||
description="Use int8 model",
|
||
disabled=ov_yolo_int8_model is None,
|
||
)
|
||
|
||
use_int8
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Checkbox(value=True, description='Use int8 model')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
WEBCAM_INFERENCE = False
|
||
|
||
if WEBCAM_INFERENCE:
|
||
VIDEO_SOURCE = 0 # Webcam
|
||
else:
|
||
download_file(
|
||
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/people.mp4",
|
||
directory="data",
|
||
)
|
||
VIDEO_SOURCE = "data/people.mp4"
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
'data/people.mp4' already exists.
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
run_object_detection(
|
||
det_model=ov_yolo_model if not use_int8.value else ov_yolo_int8_model,
|
||
source=VIDEO_SOURCE,
|
||
flip=True,
|
||
use_popup=False,
|
||
)
|
||
|
||
|
||
|
||
.. image:: yolov10-optimization-with-output_files/yolov10-optimization-with-output_50_0.png
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Source ended
|
||
|
||
|
||
Gradio Interactive Demo
|
||
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
import gradio as gr
|
||
|
||
|
||
def yolov10_inference(image, int8, conf_threshold, iou_threshold):
|
||
model = ov_yolo_model if not int8 else ov_yolo_int8_model
|
||
results = model(source=image, iou=iou_threshold, conf=conf_threshold, verbose=False)[0]
|
||
annotated_image = Image.fromarray(results.plot())
|
||
|
||
return annotated_image
|
||
|
||
|
||
with gr.Blocks() as demo:
|
||
gr.HTML(
|
||
"""
|
||
<h1 style='text-align: center'>
|
||
YOLOv10: Real-Time End-to-End Object Detection using OpenVINO
|
||
</h1>
|
||
"""
|
||
)
|
||
with gr.Row():
|
||
with gr.Column():
|
||
image = gr.Image(type="numpy", label="Image")
|
||
conf_threshold = gr.Slider(
|
||
label="Confidence Threshold",
|
||
minimum=0.1,
|
||
maximum=1.0,
|
||
step=0.1,
|
||
value=0.2,
|
||
)
|
||
iou_threshold = gr.Slider(
|
||
label="IoU Threshold",
|
||
minimum=0.1,
|
||
maximum=1.0,
|
||
step=0.1,
|
||
value=0.45,
|
||
)
|
||
use_int8 = gr.Checkbox(
|
||
value=ov_yolo_int8_model is not None,
|
||
visible=ov_yolo_int8_model is not None,
|
||
label="Use INT8 model",
|
||
)
|
||
yolov10_infer = gr.Button(value="Detect Objects")
|
||
|
||
with gr.Column():
|
||
output_image = gr.Image(type="pil", label="Annotated Image")
|
||
|
||
yolov10_infer.click(
|
||
fn=yolov10_inference,
|
||
inputs=[
|
||
image,
|
||
use_int8,
|
||
conf_threshold,
|
||
iou_threshold,
|
||
],
|
||
outputs=[output_image],
|
||
)
|
||
examples = gr.Examples(
|
||
[
|
||
"data/coco_bike.jpg",
|
||
],
|
||
inputs=[
|
||
image,
|
||
],
|
||
)
|
||
|
||
|
||
try:
|
||
demo.launch(debug=False)
|
||
except Exception:
|
||
demo.launch(debug=False, share=True)
|