natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

port-19307 (#19310) 

Porting: https://github.com/openvinotoolkit/openvino/pull/19307
Updating tutorials: adding table of contents and new notebooks.
This commit is contained in:
Sebastian Golebiewski 2023-08-21 16:47:28 +02:00 committed by GitHub
parent 54f6f11186
commit 0aa5a8f704
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
360 changed files with 19728 additions and 5810 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/nbdoc/consts.py
View File

@ -8,7 +8,7 @@ repo_owner = "openvinotoolkit"
repo_name = "openvino_notebooks"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/"
blacklisted_extensions = ['.xml', '.bin']

39
docs/notebooks/001-hello-world-with-output.rst
View File

@ -1,6 +1,8 @@
Hello Image Classification
==========================
.. _top:
This basic introduction to OpenVINO™ shows how to do inference with an
image classification model.
@ -11,10 +13,19 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ is used in
this tutorial. For more information about how OpenVINO IR models are
created, refer to the `TensorFlow to
OpenVINO <101-tensorflow-classification-to-openvino-with-output.html>`__
tutorial.
tutorial.
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Download the Model and data samples <#download-the-model-and-data-samples>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Load an Image <#load-an-image>`__
- `Do Inference <#do-inference>`__
Imports `⇑ <#top>`__
############################################
.. code:: ipython3
@ -29,8 +40,8 @@ Imports
sys.path.append("../utils")
from notebook_utils import download_file
Download the Model and data samples
-----------------------------------
Download the Model and data samples `⇑ <#top>`__
########################################################################
.. code:: ipython3
@ -63,10 +74,10 @@ Download the Model and data samples
artifacts/v3-small_224_1.0_float.bin: 0%| | 0.00/4.84M [00:00<?, ?B/s]
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
############################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -91,8 +102,8 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------
Load the Model `⇑ <#top>`__
###################################################
.. code:: ipython3
@ -102,8 +113,8 @@ Load the Model
output_layer = compiled_model.output(0)
Load an Image
-------------
Load an Image `⇑ <#top>`__
##################################################
.. code:: ipython3
@ -122,8 +133,8 @@ Load an Image
.. image:: 001-hello-world-with-output_files/001-hello-world-with-output_10_0.png
Do Inference
------------
Do Inference `⇑ <#top>`__
#################################################
.. code:: ipython3

6
docs/notebooks/001-hello-world-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/001-hello-world-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/001-hello-world-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/001-hello-world-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="001-hello-world-with-output_10_0.png">001-hello-world-with-output_10_0.png</a> 12-Jul-2023 00:11 387941
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/001-hello-world-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="001-hello-world-with-output_10_0.png">001-hello-world-with-output_10_0.png</a> 16-Aug-2023 01:31 387941
</pre><hr></body>
</html>

112
docs/notebooks/002-openvino-api-with-output.rst
View File

@ -4,29 +4,27 @@ OpenVINO™ Runtime API Tutorial
This notebook explains the basics of the OpenVINO Runtime API. It
covers:
- `Loading OpenVINO Runtime and Showing
Info <#Loading-OpenVINO-Runtime-and-Showing-Info>`__
- `Loading a Model <#Loading-a-Model>`__
- `Loading OpenVINO Runtime and Showing Info <#loading-openvino-runtime-and-showing-info>`__
- `Loading a Model <#loading-a-model>`__
- `OpenVINO IR Model <#OpenVINO-IR-Model>`__
- `ONNX Model <#ONNX-Model>`__
- `PaddlePaddle Model <#PaddlePaddle-Model>`__
- `TensorFlow Model <#TensorFlow-Model>`__
- `TensorFlow Lite Model <#TensorFlow-Lite-Model>`__
- `OpenVINO IR Model <#openvino-ir-model>`__
- `ONNX Model <#onnx-model>`__
- `PaddlePaddle Model <#paddlepaddle-model>`__
- `TensorFlow Model <#tensorflow-model>`__
- `TensorFlow Lite Model <#tensorflow-lite-model>`__
- `Getting Information about a
Model <#Getting-Information-about-a-Model>`__
- `Getting Information about a Model <#getting-information-about-a-model>`__
- `Model Inputs <#Model-Inputs>`__
- `Model Outputs <#Model-Outputs>`__
- `Model Inputs <#model-inputs>`__
- `Model Outputs <#model-outputs>`__
- `Doing Inference on a Model <#Doing-Inference-on-a-Model>`__
- `Reshaping and Resizing <#Reshaping-and-Resizing>`__
- `Doing Inference on a Model <#doing-inference-on-a-model>`__
- `Reshaping and Resizing <#reshaping-and-resizing>`__
- `Change Image Size <#Change-Image-Size>`__
- `Change Batch Size <#Change-Batch-Size>`__
- `Change Image Size <#change-image-size>`__
- `Change Batch Size <#change-batch-size>`__
- `Caching a Model <#Caching-a-Model>`__
- `Caching a Model <#caching-a-model>`__
The notebook is divided into sections with headers. The next cell
contains global requirements installation and imports. Each section is
@ -54,12 +52,12 @@ same.
.. parsed-literal::
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.65.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.5.7)
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.7.22)
Loading OpenVINO Runtime and Showing Info
@ -116,7 +114,7 @@ OpenVINO IR Model
An OpenVINO IR (Intermediate Representation) model consists of an
``.xml`` file, containing information about network topology, and a
``.bin`` file, containing the weights and biases binary data. Models in
OpenVINO IR format are obtained by using Model Optimizer tool. The
OpenVINO IR format are obtained by using model conversion API. The
``read_model()`` function expects the ``.bin`` weights file to have the
same filename and be located in the same directory as the ``.xml`` file:
``model_weights_file == Path(model_xml).with_suffix(".bin")``. If this
@ -124,16 +122,16 @@ is the case, specifying the weights file is optional. If the weights
file has a different filename, it can be specified using the ``weights``
parameter in ``read_model()``.
The OpenVINO `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html#doxid-openvino-docs-m-o-d-g-deep-learning-model-optimizer-dev-guide>`__
tool is used to convert models to OpenVINO IR format. Model Optimizer
reads the original model and creates an OpenVINO IR model (.xml and .bin
files) so inference can be performed without delays due to format
conversion. Optionally, Model Optimizer can adjust the model to be more
suitable for inference, for example, by alternating input shapes,
embedding preprocessing and cutting training parts off. For information
on how to convert your existing TensorFlow, PyTorch or ONNX model to
OpenVINO IR format with Model Optimizer, refer to the
The OpenVINO `model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html#doxid-openvino-docs-m-o-d-g-deep-learning-model-optimizer-dev-guide>`__
tool is used to convert models to OpenVINO IR format. Model conversion
API reads the original model and creates an OpenVINO IR model (``.xml``
and ``.bin`` files) so inference can be performed without delays due to
format conversion. Optionally, model conversion API can adjust the model
to be more suitable for inference, for example, by alternating input
shapes, embedding preprocessing and cutting training parts off. For
information on how to convert your existing TensorFlow, PyTorch or ONNX
model to OpenVINO IR format with model conversion API, refer to the
`tensorflow-to-openvino <101-tensorflow-classification-to-openvino-with-output.html>`__
and
`pytorch-onnx-to-openvino <102-pytorch-onnx-to-openvino-with-output.html>`__
@ -165,7 +163,7 @@ notebooks.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -212,7 +210,7 @@ points to the filename of an ONNX model.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx')
@ -268,7 +266,7 @@ without any conversion step. Pass the filename with extension to
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams')
@ -292,13 +290,15 @@ TensorFlow Model
~~~~~~~~~~~~~~~~
TensorFlow models saved in frozen graph format can also be passed to
``read_model`` starting in OpenVINO 2022.3.
.. note::
* Directly loading TensorFlow models is available as a preview feature in the OpenVINO 2022.3 release. Fully functional support will be provided in the upcoming 2023 releases.
* Currently support is limited to only frozen graph inference format. Other TensorFlow model formats must be converted to OpenVINO IR using `Model Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
``read_model`` starting in OpenVINO 2022.3.
**NOTE**: Directly loading TensorFlow models is available as a
preview feature in the OpenVINO 2022.3 release. Fully functional
support will be provided in the upcoming 2023 releases. Currently
support is limited to only frozen graph inference format. Other
TensorFlow model formats must be converted to OpenVINO IR using
`model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
.. code:: ipython3
@ -318,7 +318,7 @@ TensorFlow models saved in frozen graph format can also be passed to
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb')
@ -370,7 +370,7 @@ It is pre-trained model optimized to work with TensorFlow Lite.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite')
@ -395,7 +395,7 @@ Getting Information about a Model
The OpenVINO Model instance stores information about the model.
Information about the inputs and outputs of the model are in
``model.inputs`` and ``model.outputs``. These are also properties of the
CompiledModel instance. While using ``model.inputs`` and
``CompiledModel`` instance. While using ``model.inputs`` and
``model.outputs`` in the cells below, you can also use
``compiled_model.inputs`` and ``compiled_model.outputs``.
@ -419,7 +419,7 @@ CompiledModel instance. While using ``model.inputs`` and
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -581,8 +581,8 @@ on a model, first create an inference request by calling the
``compiled_model`` that was loaded with ``compile_model()``. Then, call
the ``infer()`` method of ``InferRequest``. It expects one argument:
``inputs``. This is a dictionary that maps input layer names to input
data or list of input data in np.ndarray format, where the position of
the input tensor corresponds to input index. If a model has a single
data or list of input data in ``np.ndarray`` format, where the position
of the input tensor corresponds to input index. If a model has a single
input, wrapping to a dictionary or list can be omitted.
.. code:: ipython3
@ -612,7 +612,7 @@ input, wrapping to a dictionary or list can be omitted.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -707,10 +707,10 @@ add the ``N`` dimension (where ``N``\ = 1) by calling the
**Do inference**
Now that the input data is in the right shape, run inference. The
CompiledModel inference result is a dictionary where keys are the Output
class instances (the same keys in ``compiled_model.outputs`` that can
also be obtained with ``compiled_model.output(index)``) and values -
predicted result in np.array format.
``CompiledModel`` inference result is a dictionary where keys are the
Output class instances (the same keys in ``compiled_model.outputs`` that
can also be obtained with ``compiled_model.output(index)``) and values -
predicted result in ``np.array`` format.
.. code:: ipython3
@ -797,7 +797,7 @@ input shape.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin')
@ -948,7 +948,7 @@ the cache.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')

60
docs/notebooks/003-hello-segmentation-with-output.rst
View File

@ -1,6 +1,8 @@
Hello Image Segmentation
========================
.. _top:
A very basic introduction to using segmentation models with OpenVINO™.
In this tutorial, a pre-trained
@ -10,8 +12,19 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ is used.
ADAS stands for Advanced Driver Assistance Services. The model
recognizes four classes: background, road, curb and mark.
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Download model weights <#download-model-weights>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Load an Image <#load-an-image>`__
- `Do Inference <#do-inference>`__
- `Prepare Data for Visualization <#prepare-data-for-visualization>`__
- `Visualize data <#visualize-data>`__
Imports `⇑ <#top>`__
#########################################
.. code:: ipython3
@ -24,8 +37,9 @@ Imports
sys.path.append("../utils")
from notebook_utils import segmentation_map_to_image, download_file
Download model weights
----------------------
Download model weights `⇑ <#top>`__
#############################################################################################################################
.. code:: ipython3
@ -61,10 +75,11 @@ Download model weights
model/road-segmentation-adas-0001.bin: 0%| | 0.00/720k [00:00<?, ?B/s]
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
#############################################################################################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -89,8 +104,9 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------
Load the Model `⇑ <#top>`__
#############################################################################################################################
.. code:: ipython3
@ -102,11 +118,10 @@ Load the Model
input_layer_ir = compiled_model.input(0)
output_layer_ir = compiled_model.output(0)
Load an Image
-------------
Load an Image `⇑ <#top>`__
#############################################################################################################################
A sample image from the `Mapillary
Vistas <https://www.mapillary.com/dataset/vistas>`__ dataset is
A sample image from the `Mapillary Vistas <https://www.mapillary.com/dataset/vistas>`__ dataset is
provided.
.. code:: ipython3
@ -134,7 +149,7 @@ provided.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f021436e0a0>
<matplotlib.image.AxesImage at 0x7f1f2c248430>
@ -142,8 +157,9 @@ provided.
.. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_10_1.png
Do Inference
------------
Do Inference `⇑ <#top>`__
#############################################################################################################################
.. code:: ipython3
@ -159,7 +175,7 @@ Do Inference
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f02142bce50>
<matplotlib.image.AxesImage at 0x7f1f2c18bfd0>
@ -167,8 +183,9 @@ Do Inference
.. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_12_1.png
Prepare Data for Visualization
------------------------------
Prepare Data for Visualization `⇑ <#top>`__
#############################################################################################################################
.. code:: ipython3
@ -185,8 +202,9 @@ Prepare Data for Visualization
# Create an image with mask.
image_with_mask = cv2.addWeighted(resized_mask, alpha, rgb_image, 1 - alpha, 0)
Visualize data
--------------
Visualize data `⇑ <#top>`__
#############################################################################################################################
.. code:: ipython3

10
docs/notebooks/003-hello-segmentation-with-output_files/index.html
View File

@ -1,9 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/003-hello-segmentation-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/003-hello-segmentation-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/003-hello-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="003-hello-segmentation-with-output_10_1.png">003-hello-segmentation-with-output_10_1.png</a> 12-Jul-2023 00:11 249032
<a href="003-hello-segmentation-with-output_12_1.png">003-hello-segmentation-with-output_12_1.png</a> 12-Jul-2023 00:11 20550
<a href="003-hello-segmentation-with-output_16_0.png">003-hello-segmentation-with-output_16_0.png</a> 12-Jul-2023 00:11 260045
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/003-hello-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="003-hello-segmentation-with-output_10_1.png">003-hello-segmentation-with-output_10_1.png</a> 16-Aug-2023 01:31 249032
<a href="003-hello-segmentation-with-output_12_1.png">003-hello-segmentation-with-output_12_1.png</a> 16-Aug-2023 01:31 20550
<a href="003-hello-segmentation-with-output_16_0.png">003-hello-segmentation-with-output_16_0.png</a> 16-Aug-2023 01:31 260045
</pre><hr></body>
</html>

44
docs/notebooks/004-hello-detection-with-output.rst
View File

@ -1,6 +1,8 @@
Hello Object Detection
======================
.. _top:
A very basic introduction to using object detection models with
OpenVINO™.
@ -14,10 +16,20 @@ shape of ``[100, 5]``. Each detected text box is stored in the
``(x_min, y_min)`` are the coordinates of the top left bounding box
corner, ``(x_max, y_max)`` are the coordinates of the bottom right
bounding box corner and ``conf`` is the confidence for the predicted
class.
class.
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Download model weights <#download-model-weights>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Load an Image <#load-an-image>`__
- `Do Inference <#do-inference>`__
- `Visualize Results <#visualize-results>`__
Imports `⇑ <#top>`__
########################################
.. code:: ipython3
@ -31,8 +43,8 @@ Imports
sys.path.append("../utils")
from notebook_utils import download_file
Download model weights
----------------------
Download model weights `⇑ <#top>`__
#######################################################
.. code:: ipython3
@ -67,10 +79,10 @@ Download model weights
model/horizontal-text-detection-0001.bin: 0%| | 0.00/7.39M [00:00<?, ?B/s]
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
###########################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -95,8 +107,8 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------
Load the Model `⇑ <#top>`__
###############################################
.. code:: ipython3
@ -108,8 +120,8 @@ Load the Model
input_layer_ir = compiled_model.input(0)
output_layer_ir = compiled_model.output("boxes")
Load an Image
-------------
Load an Image `⇑ <#top>`__
##############################################
.. code:: ipython3
@ -132,8 +144,8 @@ Load an Image
.. image:: 004-hello-detection-with-output_files/004-hello-detection-with-output_10_0.png
Do Inference
------------
Do Inference `⇑ <#top>`__
##############################################
.. code:: ipython3
@ -143,8 +155,8 @@ Do Inference
# Remove zero only boxes.
boxes = boxes[~np.all(boxes == 0, axis=1)]
Visualize Results
-----------------
Visualize Results `⇑ <#top>`__
##################################################
.. code:: ipython3

8
docs/notebooks/004-hello-detection-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/004-hello-detection-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/004-hello-detection-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/004-hello-detection-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="004-hello-detection-with-output_10_0.png">004-hello-detection-with-output_10_0.png</a> 12-Jul-2023 00:11 305482
<a href="004-hello-detection-with-output_15_0.png">004-hello-detection-with-output_15_0.png</a> 12-Jul-2023 00:11 457214
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/004-hello-detection-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="004-hello-detection-with-output_10_0.png">004-hello-detection-with-output_10_0.png</a> 16-Aug-2023 01:31 305482
<a href="004-hello-detection-with-output_15_0.png">004-hello-detection-with-output_15_0.png</a> 16-Aug-2023 01:31 457214
</pre><hr></body>
</html>

135
docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst
View File

@ -1,18 +1,42 @@
Convert a TensorFlow Model to OpenVINO™
=======================================
This short tutorial shows how to convert a TensorFlow
`MobileNetV3 <https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
image classification model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
.. _top:
| This short tutorial shows how to convert a TensorFlow
`MobileNetV3 <https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
image classification model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using `model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
| **Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Download model <#download-model>`__
- `Convert a Model to OpenVINO IR Format <#convert-a-model-to-openvino-ir-format>`__
- `Convert a TensorFlow Model to OpenVINO IR Format <#convert-a-tensorflow-model-to-openvino-ir-format>`__
- `Test Inference on the Converted Model <#test-inference-on-the-converted-model>`__
- `Load the Model <#load-the-model>`__
- `Select inference device <#select-inference-device>`__
- `Get Model Information <#get-model-information>`__
- `Load an Image <#load-an-image>`__
- `Do Inference <#do-inference>`__
- `Timing <#timing>`__
Imports `⇑ <#top>`__
###############################################################################################################################
Imports
-------
.. code:: ipython3
@ -29,14 +53,15 @@ Imports
.. parsed-literal::
2023-07-11 22:24:31.659014: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:24:31.694466: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:26:34.199621: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:26:34.233464: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:24:32.210417: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:26:34.746193: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -48,8 +73,9 @@ Settings
ir_path = Path("model/v3-small_224_1.0_float.xml")
Download model
--------------
Download model `⇑ <#top>`__
###############################################################################################################################
Load model using `tf.keras.applications
api <https://www.tensorflow.org/api_docs/python/tf/keras/applications/MobileNetV3Small>`__
@ -68,7 +94,7 @@ and save it to the disk.
.. parsed-literal::
2023-07-11 22:24:33.117393: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
2023-08-15 22:26:35.659386: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
@ -79,9 +105,9 @@ and save it to the disk.
.. parsed-literal::
2023-07-11 22:24:37.315661: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
2023-08-15 22:26:39.846021: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
[[{{node inputs}}]]
2023-07-11 22:24:40.473876: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
2023-08-15 22:26:42.992490: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
[[{{node inputs}}]]
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 54). These functions will not be directly callable after loading.
@ -96,25 +122,27 @@ and save it to the disk.
INFO:tensorflow:Assets written to: model/v3-small_224_1.0_float/assets
Convert a Model to OpenVINO IR Format
-------------------------------------
Convert a Model to OpenVINO IR Format `⇑ <#top>`__
###############################################################################################################################
Convert a TensorFlow Model to OpenVINO IR Format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Call the OpenVINO Model Optimizer Python API to convert the TensorFlow
model to OpenVINO IR. ``mo.convert_model`` function accept path to saved
Convert a TensorFlow Model to OpenVINO IR Format `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use the model conversion Python API to convert the TensorFlow model to
OpenVINO IR. The ``mo.convert_model`` function accept path to saved
model directory and returns OpenVINO Model class instance which
represents this model. Obtained model is ready to use and loading on
device using ``compile_model`` or can be saved on disk using
``serialize`` function. See the `Model Optimizer Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__
for more information about Model Optimizer and TensorFlow models
conversion.
represents this model. Obtained model is ready to use and to be loaded
on a device using ``compile_model`` or can be saved on a disk using the
``serialize`` function. See the
`tutorial <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__
for more information about using model conversion API with TensorFlow
models.
.. code:: ipython3
# Run Model Optimizer if the IR model file does not exist
# Run model conversion API if the IR model file does not exist
if not ir_path.exists():
print("Exporting TensorFlow model to IR... This may take a few minutes.")
ov_model = mo.convert_model(saved_model_dir=model_path, input_shape=[[1, 224, 224, 3]], compress_to_fp16=True)
@ -128,21 +156,24 @@ conversion.
Exporting TensorFlow model to IR... This may take a few minutes.
Test Inference on the Converted Model
-------------------------------------
Test Inference on the Converted Model `⇑ <#top>`__
###############################################################################################################################
Load the Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load the Model
~~~~~~~~~~~~~~
.. code:: ipython3
core = Core()
model = core.read_model(ir_path)
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -170,8 +201,9 @@ select device from dropdown list for running inference using OpenVINO
compiled_model = core.compile_model(model=model, device_name=device.value)
Get Model Information
~~~~~~~~~~~~~~~~~~~~~
Get Model Information `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -179,8 +211,9 @@ Get Model Information
output_key = compiled_model.output(0)
network_input_shape = input_key.shape
Load an Image
~~~~~~~~~~~~~
Load an Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load an image, resize it, and convert it to the input shape of the
network.
@ -203,8 +236,9 @@ network.
.. image:: 101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_18_0.png
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -228,8 +262,9 @@ Do Inference
Timing
------
Timing `⇑ <#top>`__
###############################################################################################################################
Measure the time it takes to do inference on thousand images. This gives
an indication of performance. For more accurate benchmarking, use the
@ -258,5 +293,5 @@ performance.
.. parsed-literal::
IR model in OpenVINO Runtime/CPU: 0.0010 seconds per image, FPS: 998.88
IR model in OpenVINO Runtime/CPU: 0.0010 seconds per image, FPS: 988.20

6
docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="101-tensorflow-classification-to-openvino-with-output_18_0.png">101-tensorflow-classification-to-openvino-with-..&gt;</a> 12-Jul-2023 00:11 387941
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="101-tensorflow-classification-to-openvino-with-output_18_0.png">101-tensorflow-classification-to-openvino-with-..&gt;</a> 16-Aug-2023 01:31 387941
</pre><hr></body>
</html>

133
docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst
View File

@ -1,6 +1,8 @@
Convert a PyTorch Model to ONNX and OpenVINO™ IR
================================================
.. _top:
This tutorial demonstrates step-by-step instructions on how to do
inference on a PyTorch semantic segmentation model, using OpenVINO
Runtime.
@ -28,16 +30,43 @@ all 80 classes, the segmentation model has been trained on 20 classes
from the `PASCAL VOC <http://host.robots.ox.ac.uk/pascal/VOC/>`__
dataset: **background, aeroplane, bicycle, bird, boat, bottle, bus, car,
cat, chair, cow, dining table, dog, horse, motorbike, person, potted
plant, sheep, sofa, train, tvmonitor**
plant, sheep, sofa, train, tv monitor**
More information about the model is available in the `torchvision
documentation <https://pytorch.org/vision/main/models/lraspp.html>`__
Preparation
-----------
**Table of contents**:
Imports
~~~~~~~
- `Preparation <#preparation>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Load Model <#load-model>`__
- `ONNX Model Conversion <#onnx-model-conversion>`__
- `Convert PyTorch model to ONNX <#convert-pytorch-model-to-onnx>`__
- `Convert ONNX Model to OpenVINO IR Format <#convert-onnx-model-to-openvino-ir-format>`__
- `Show Results <#show-results>`__
- `Load and Preprocess an Input Image <#load-and-preprocess-an-input-image>`__
- `Load the OpenVINO IR Network and Run Inference on the ONNX model <#load-the-openvino-ir-network-and-run-inference-on-the-onnx-model>`__
- `1. ONNX Model in OpenVINO Runtime <#onnx-model-in-openvino-runtime>`__
- `Select an inference device <#select-an-inference-device>`__
- `2. OpenVINO IR Model in OpenVINO Runtime <#openvino-ir-model-in-openvino-runtime>`__
- `Select the inference device <#select-the-inference-device>`__
- `PyTorch Comparison <#pytorch-comparison>`__
- `Performance Comparison <#performance-comparison>`__
- `References <#references>`__
Preparation `⇑ <#top>`__
########################################################################
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -55,8 +84,8 @@ Imports
sys.path.append("../utils")
from notebook_utils import segmentation_map_to_image, viz_result_image, SegmentationMap, Label, download_file
Settings
~~~~~~~~
Settings `⇑ <#top>`__
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Set a name for the model, then define width and height of the image that
will be used by the network during inference. According to the input
@ -77,16 +106,15 @@ transforms function, the model is pre-trained on images with a height of
onnx_path.parent.mkdir()
ir_path = onnx_path.with_suffix(".xml")
Load Model
~~~~~~~~~~
Load Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Generally, PyTorch models represent an instance of ``torch.nn.Module``
class, initialized by a state dictionary with model weights. Typical
steps for getting a pre-trained model:
1. Create instance of model class
steps for getting a pre-trained model: 1. Create instance of model class
2. Load checkpoint state dict, which contains pre-trained model weights
3. Turn model to evaluation for switching some operations to inference mode
3. Turn model to evaluation for switching some operations to inference
mode
The ``torchvision`` module provides a ready to use set of functions for
model class initialization. We will use
@ -129,11 +157,11 @@ have not downloaded the model before.
Loaded PyTorch LRASPP MobileNetV3 model
ONNX Model Conversion
---------------------
ONNX Model Conversion `⇑ <#top>`__
################################################################################
Convert PyTorch model to ONNX
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Convert PyTorch model to ONNX `⇑ <#top>`__
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO supports PyTorch models that are exported in ONNX format. We
will use the ``torch.onnx.export`` function to obtain the ONNX model,
@ -172,14 +200,13 @@ line of the output will read:
ONNX model exported to model/lraspp_mobilenet_v3_large.onnx.
Convert ONNX Model to OpenVINO IR Format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Convert ONNX Model to OpenVINO IR Format `⇑ <#top>`__
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use Model Optimizer to convert the ONNX model to OpenVINO IR with
``FP16`` precision. The models are saved inside the current directory.
For more information about Model Optimizer, see the `Model Optimizer
Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
To convert the ONNX model to OpenVINO IR with ``FP16`` precision, use
model conversion API. The models are saved inside the current directory.
For more information on how to convert models, see this
`page <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
.. code:: ipython3
@ -199,14 +226,14 @@ Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_O
Exporting ONNX model to IR... This may take a few minutes.
Show Results
------------
Show Results `⇑ <#top>`__
######################################################################
Confirm that the segmentation results look as expected by comparing
model predictions on the ONNX, OpenVINO IR and PyTorch models.
Load and Preprocess an Input Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load and Preprocess an Input Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Images need to be normalized before propagating through the network.
@ -237,8 +264,8 @@ Images need to be normalized before propagating through the network.
input_image = np.expand_dims(np.transpose(resized_image, (2, 0, 1)), 0)
normalized_input_image = np.expand_dims(np.transpose(normalized_image, (2, 0, 1)), 0)
Load the OpenVINO IR Network and Run Inference on the ONNX model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the OpenVINO IR Network and Run Inference on the ONNX model `⇑ <#top>`__
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO Runtime can load ONNX models directly. First, load the ONNX
model, do inference and show the results. Then, load the model that was
@ -246,8 +273,8 @@ converted to OpenVINO Intermediate Representation (OpenVINO IR) with
Model Optimizer and do inference on that model, and show the results on
an image.
1. ONNX Model in OpenVINO Runtime
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. ONNX Model in OpenVINO Runtime `⇑ <#top>`__
------------------------------------------------------------------------------------------
.. code:: ipython3
@ -257,10 +284,10 @@ an image.
# Read model to OpenVINO Runtime
model_onnx = core.read_model(model=onnx_path)
Select inference device
^^^^^^^^^^^^^^^^^^^^^^^
Select an inference device `⇑ <#top>`__
...................................................................................
select device from dropdown list for running inference using OpenVINO
Select a device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -339,13 +366,13 @@ be applied to each label for more convenient visualization.
2. OpenVINO IR Model in OpenVINO Runtime
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2. OpenVINO IR Model in OpenVINO Runtime `⇑ <#top>`__
------------------------------------------------------------------------------------------------------
Select inference device
^^^^^^^^^^^^^^^^^^^^^^^
Select the inference device `⇑ <#top>`__
.....................................................................................
select device from dropdown list for running inference using OpenVINO
Select a device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -389,8 +416,8 @@ select device from dropdown list for running inference using OpenVINO
PyTorch Comparison
------------------
PyTorch Comparison `⇑ <#top>`__
############################################################################
Do inference on the PyTorch model to verify that the output visually
looks the same as the output on the ONNX/OpenVINO IR models.
@ -415,8 +442,8 @@ looks the same as the output on the ONNX/OpenVINO IR models.
Performance Comparison
----------------------
Performance Comparison `⇑ <#top>`__
################################################################################
Measure the time it takes to do inference on twenty images. This gives
an indication of performance. For more accurate benchmarking, use the
@ -488,9 +515,9 @@ performance.
.. parsed-literal::
PyTorch model on CPU: 0.039 seconds per image, FPS: 25.80
ONNX model in OpenVINO Runtime/CPU: 0.031 seconds per image, FPS: 31.95
OpenVINO IR model in OpenVINO Runtime/CPU: 0.031 seconds per image, FPS: 32.67
PyTorch model on CPU: 0.037 seconds per image, FPS: 27.19
ONNX model in OpenVINO Runtime/CPU: 0.031 seconds per image, FPS: 32.33
OpenVINO IR model in OpenVINO Runtime/CPU: 0.032 seconds per image, FPS: 31.72
**Show Device Information**
@ -508,8 +535,8 @@ performance.
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
References
----------
References `⇑ <#top>`__
######################################################################
- `Torchvision <https://pytorch.org/vision/stable/index.html>`__
- `Pytorch ONNX
@ -517,7 +544,7 @@ References
- `PIP install openvino-dev <https://pypi.org/project/openvino-dev/>`__
- `OpenVINO ONNX
support <https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_ONNX_Support.html>`__
- `Model Optimizer
Documentation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__
- `Model Optimizer Pytorch conversion
guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_PyTorch.html>`__
- `Model Conversion API
documentation <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
- `Converting Pytorch
model <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_PyTorch.html>`__

10
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html
View File

@ -1,9 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-onnx-to-openvino-with-output_21_0.png">102-pytorch-onnx-to-openvino-with-output_21_0.png</a> 12-Jul-2023 00:11 465692
<a href="102-pytorch-onnx-to-openvino-with-output_26_0.png">102-pytorch-onnx-to-openvino-with-output_26_0.png</a> 12-Jul-2023 00:11 465695
<a href="102-pytorch-onnx-to-openvino-with-output_28_0.png">102-pytorch-onnx-to-openvino-with-output_28_0.png</a> 12-Jul-2023 00:11 465692
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-onnx-to-openvino-with-output_21_0.png">102-pytorch-onnx-to-openvino-with-output_21_0.png</a> 16-Aug-2023 01:31 465692
<a href="102-pytorch-onnx-to-openvino-with-output_26_0.png">102-pytorch-onnx-to-openvino-with-output_26_0.png</a> 16-Aug-2023 01:31 465695
<a href="102-pytorch-onnx-to-openvino-with-output_28_0.png">102-pytorch-onnx-to-openvino-with-output_28_0.png</a> 16-Aug-2023 01:31 465692
</pre><hr></body>
</html>

182
docs/notebooks/102-pytorch-to-openvino-with-output.rst
View File

@ -1,6 +1,8 @@
Convert a PyTorch Model to OpenVINO™ IR
=======================================
.. _top:
This tutorial demonstrates step-by-step instructions on how to do
inference on a PyTorch classification model using OpenVINO Runtime.
Starting from OpenVINO 2023.0 release, OpenVINO supports direct PyTorch
@ -27,12 +29,45 @@ network design spaces that parametrize populations of networks. The
overall process is analogous to the classic manual design of networks
but elevated to the design space level. The RegNet design space provides
simple and fast networks that work well across a wide range of flop
regimes.
regimes.
Prerequisites
-------------
**Table of contents**:
Install notebook dependecies
- `Prerequisites <#prerequisites>`__
- `Load PyTorch Model <#load-pytorch-model>`__
- `Prepare Input Data <#prepare-input-data>`__
- `Run PyTorch Model Inference <#run-pytorch-model-inference>`__
- `Benchmark PyTorch Model Inference <#benchmark-pytorch-model-inference>`__
- `Convert PyTorch Model to OpenVINO Intermediate Representation <#convert-pytorch-model-to-openvino-intermediate-representation>`__
- `Select inference device <#select-inference-device>`__
- `Run OpenVINO Model Inference <#run-openvino-model-inference>`__
- `Benchmark OpenVINO Model Inference <#benchmark-openvino-model-inference>`__
- `Convert PyTorch Model with Static Input Shape <#convert-pytorch-model-with-static-input-shape>`__
- `Select inference device <#select-inference-device>`__
- `Run OpenVINO Model Inference with Static Input Shape <#run-openvino-model-inference-with-static-input-shape>`__
- `Benchmark OpenVINO Model Inference with Static Input Shape <#benchmark-openvino-model-inference-with-static-input-shape>`__
- `Convert TorchScript Model to OpenVINO Intermediate Representation <#convert-torchscript-model-to-openvino-intermediate-representation>`__
- `Scripted Model <#scripted-model>`__
- `Benchmark Scripted Model Inference <#benchmark-scripted-model-inference>`__
- `Convert PyTorch Scripted Model to OpenVINO Intermediate Representation <#convert-pytorch-scripted-model-to-openvino-intermediate-representation>`__
- `Benchmark OpenVINO Model Inference Converted From Scripted Model <#benchmark-openvino-model-inference-converted-from-scripted-model>`__
- `Traced Model <#traced-model>`__
- `Benchmark Traced Model Inference <#benchmark-traced-model-inference>`__
- `Convert PyTorch Traced Model to OpenVINO Intermediate Representation <#convert-pytorch-traced-model-to-openvino-intermediate-representation>`__
- `Benchmark OpenVINO Model Inference Converted From Traced Model <#benchmark-openvino-model-inference-converted-from-traced-model>`__
Prerequisites `⇑ <#top>`__
###############################################################################################################################
Install notebook dependencies
.. code:: ipython3
@ -64,8 +99,9 @@ Download input data and label map
imagenet_classes = labels_file.open("r").read().splitlines()
Load PyTorch Model
------------------
Load PyTorch Model `⇑ <#top>`__
###############################################################################################################################
Generally, PyTorch models represent an instance of the
``torch.nn.Module`` class, initialized by a state dictionary with model
@ -73,7 +109,8 @@ weights. Typical steps for getting a pre-trained model:
1. Create an instance of a model class
2. Load checkpoint state dict, which contains pre-trained model weights
3. Turn the model to evaluation for switching some operations to inference mode
3. Turn the model to evaluation for switching some operations to
inference mode
The ``torchvision`` module provides a ready-to-use set of functions for
model class initialization. We will use
@ -94,8 +131,9 @@ enum ``RegNet_Y_800MF_Weights.DEFAULT``.
# switch model to inference mode
model.eval();
Prepare Input Data
~~~~~~~~~~~~~~~~~~
Prepare Input Data `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The code below demonstrates how to preprocess input data using a
model-specific transforms module from ``torchvision``. After
@ -116,8 +154,9 @@ the first dimension.
# Add batch dimension to image tensor
input_tensor = img_transformed.unsqueeze(0)
Run PyTorch Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run PyTorch Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The model returns a vector of probabilities in raw logits format,
softmax can be applied to get normalized values in the [0, 1] range. For
@ -172,8 +211,9 @@ can be reused later.
5: hamper - 2.35%
Benchmark PyTorch Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmark PyTorch Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -185,11 +225,11 @@ Benchmark PyTorch Model Inference
.. parsed-literal::
13.5 ms ± 3.76 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
13.2 ms ± 27.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model to OpenVINO Intermediate Representation
-------------------------------------------------------------
Convert PyTorch Model to OpenVINO Intermediate Representation. `⇑ <#top>`__
###############################################################################################################################
Starting from the 2023.0 release OpenVINO supports direct PyTorch models
conversion to OpenVINO Intermediate Representation (IR) format. Model
@ -211,13 +251,17 @@ device using ``core.compile_model`` or save on disk for next usage using
``openvino.runtime.serialize``. Optionally, we can provide additional
parameters, such as:
* ``compress_to_fp16`` - flag to perform model weights compression into FP16 data format. It may reduce the required space for model storage on disk and give speedup for inference devices, where FP16 calculation is supported.
* ``example_input`` - input data sample which can be used for model tracing.
* ``input_shape`` - the shape of input tensor for conversion
- ``compress_to_fp16`` - flag to perform model weights compression into
FP16 data format. It may reduce the required space for model storage
on disk and give speedup for inference devices, where FP16
calculation is supported.
- ``example_input`` - input data sample which can be used for model
tracing.
- ``input_shape`` - the shape of input tensor for conversion
and any other advanced options supported by Model Optimizer Python API.
and any other advanced options supported by model conversion Python API.
More details can be found on this
`page <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
`page <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__
.. code:: ipython3
@ -250,10 +294,11 @@ More details can be found on this
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -298,8 +343,9 @@ select device from dropdown list for running inference using OpenVINO
Run OpenVINO Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run OpenVINO Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -329,8 +375,9 @@ Run OpenVINO Model Inference
5: hamper - 2.35%
Benchmark OpenVINO Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmark OpenVINO Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -341,11 +388,12 @@ Benchmark OpenVINO Model Inference
.. parsed-literal::
3.03 ms ± 5.57 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.03 ms ± 45.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model with Static Input Shape
---------------------------------------------
Convert PyTorch Model with Static Input Shape `⇑ <#top>`__
###############################################################################################################################
The default conversion path preserves dynamic input shapes, in order if
you want to convert the model with static shapes, you can explicitly
@ -377,10 +425,11 @@ reshaping example please check the following
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -420,8 +469,9 @@ Now, we can see that input of our converted model is tensor of shape [1,
3, 224, 224] instead of [?, 3, ?, ?] reported by previously converted
model.
Run OpenVINO Model Inference with Static Input Shape
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run OpenVINO Model Inference with Static Input Shape `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -452,7 +502,7 @@ Run OpenVINO Model Inference with Static Input Shape
Benchmark OpenVINO Model Inference with Static Input Shape
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`⇑ <#top>`__
.. code:: ipython3
@ -463,11 +513,11 @@ Benchmark OpenVINO Model Inference with Static Input Shape
.. parsed-literal::
2.79 ms ± 26.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.77 ms ± 12.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert TorchScript Model to OpenVINO Intermediate Representation
-----------------------------------------------------------------
Convert TorchScript Model to OpenVINO Intermediate Representation. `⇑ <#top>`__
###############################################################################################################################
TorchScript is a way to create serializable and optimizable models from
PyTorch code. Any TorchScript program can be saved from a Python process
@ -487,8 +537,9 @@ There are 2 possible ways to convert the PyTorch model to TorchScript:
Let’s consider both approaches and their conversion into OpenVINO IR.
Scriped Model
~~~~~~~~~~~~~
Scripted Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
``torch.jit.script`` inspects model source code and compiles it to
``ScriptModule``. After compilation model can be used for inference or
@ -540,8 +591,9 @@ Reference <https://pytorch.org/docs/stable/jit_language_reference.html#language-
5: hamper - 2.35%
Benchmark Scripted Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmark Scripted Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -552,14 +604,12 @@ Benchmark Scripted Model Inference
.. parsed-literal::
12.7 ms ± 13.4 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
12.6 ms ± 17.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Convert PyTorch Scripted Model to OpenVINO Intermediate Representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The conversion step for the scripted model to OpenVINO IR is similar to
the original PyTorch model.
Convert PyTorch Scripted Model to OpenVINO Intermediate
Representation `⇑ <#top>`__ The conversion step for the scripted model to
OpenVINO IR is similar to the original PyTorch model.
.. code:: ipython3
@ -596,7 +646,7 @@ the original PyTorch model.
Benchmark OpenVINO Model Inference Converted From Scripted Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`⇑ <#top>`__
.. code:: ipython3
@ -607,11 +657,12 @@ Benchmark OpenVINO Model Inference Converted From Scripted Model
.. parsed-literal::
3.1 ms ± 4.02 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.07 ms ± 5.58 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Traced Model
~~~~~~~~~~~~
Traced Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Using ``torch.jit.trace``, you can turn an existing module or Python
function into a TorchScript ``ScriptFunction`` or ``ScriptModule``. You
@ -619,10 +670,10 @@ must provide example inputs, and model will be executed, recording the
operations performed on all the tensors.
- The resulting recording of a standalone function produces
ScriptFunction.
``ScriptFunction``.
- The resulting recording of nn.Module.forward or nn.Module produces
ScriptModule.
- The resulting recording of ``nn.Module.forward`` or ``nn.Module``
produces ``ScriptModule``.
In the same way like scripted model, traced model can be used for
inference or saved on disk using ``torch.jit.save`` function and after
@ -667,8 +718,9 @@ original PyTorch model code definitions.
5: hamper - 2.35%
Benchmark Traced Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmark Traced Model Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -679,14 +731,12 @@ Benchmark Traced Model Inference
.. parsed-literal::
12.7 ms ± 39.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
12.7 ms ± 61.1 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Convert PyTorch Traced Model to OpenVINO Intermediate Representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The conversion step for a traced model to OpenVINO IR is similar to the
original PyTorch model.
`⇑ <#top>`__ The conversion step for a traced model to OpenVINO IR is
similar to the original PyTorch model.
.. code:: ipython3
@ -723,7 +773,7 @@ original PyTorch model.
Benchmark OpenVINO Model Inference Converted From Traced Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`⇑ <#top>`__
.. code:: ipython3
@ -734,5 +784,5 @@ Benchmark OpenVINO Model Inference Converted From Traced Model
.. parsed-literal::
3.08 ms ± 9.07 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.05 ms ± 6.85 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

32
docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html
View File

@ -1,20 +1,20 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/102-pytorch-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/102-pytorch-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/102-pytorch-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-to-openvino-with-output_11_0.jpg">102-pytorch-to-openvino-with-output_11_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_11_0.png">102-pytorch-to-openvino-with-output_11_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_20_0.jpg">102-pytorch-to-openvino-with-output_20_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_20_0.png">102-pytorch-to-openvino-with-output_20_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_31_0.jpg">102-pytorch-to-openvino-with-output_31_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_31_0.png">102-pytorch-to-openvino-with-output_31_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_35_0.jpg">102-pytorch-to-openvino-with-output_35_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_35_0.png">102-pytorch-to-openvino-with-output_35_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_39_0.jpg">102-pytorch-to-openvino-with-output_39_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_39_0.png">102-pytorch-to-openvino-with-output_39_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_43_0.jpg">102-pytorch-to-openvino-with-output_43_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_43_0.png">102-pytorch-to-openvino-with-output_43_0.png</a> 12-Jul-2023 00:11 542516
<a href="102-pytorch-to-openvino-with-output_47_0.jpg">102-pytorch-to-openvino-with-output_47_0.jpg</a> 12-Jul-2023 00:11 54874
<a href="102-pytorch-to-openvino-with-output_47_0.png">102-pytorch-to-openvino-with-output_47_0.png</a> 12-Jul-2023 00:11 542516
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/102-pytorch-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-to-openvino-with-output_11_0.jpg">102-pytorch-to-openvino-with-output_11_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_11_0.png">102-pytorch-to-openvino-with-output_11_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_20_0.jpg">102-pytorch-to-openvino-with-output_20_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_20_0.png">102-pytorch-to-openvino-with-output_20_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_31_0.jpg">102-pytorch-to-openvino-with-output_31_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_31_0.png">102-pytorch-to-openvino-with-output_31_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_35_0.jpg">102-pytorch-to-openvino-with-output_35_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_35_0.png">102-pytorch-to-openvino-with-output_35_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_39_0.jpg">102-pytorch-to-openvino-with-output_39_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_39_0.png">102-pytorch-to-openvino-with-output_39_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_43_0.jpg">102-pytorch-to-openvino-with-output_43_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_43_0.png">102-pytorch-to-openvino-with-output_43_0.png</a> 16-Aug-2023 01:31 542516
<a href="102-pytorch-to-openvino-with-output_47_0.jpg">102-pytorch-to-openvino-with-output_47_0.jpg</a> 16-Aug-2023 01:31 54874
<a href="102-pytorch-to-openvino-with-output_47_0.png">102-pytorch-to-openvino-with-output_47_0.png</a> 16-Aug-2023 01:31 542516
</pre><hr></body>
</html>

93
docs/notebooks/103-paddle-to-openvino-classification-with-output.rst
View File

@ -1,12 +1,14 @@
Convert a PaddlePaddle Model to OpenVINO™ IR
============================================
.. _top:
This notebook shows how to convert a MobileNetV3 model from
`PaddleHub <https://github.com/PaddlePaddle/PaddleHub>`__, pre-trained
on the `ImageNet <https://www.image-net.org>`__ dataset, to OpenVINO IR.
It also shows how to perform classification inference on a sample image,
using `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
Runtime <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and compares the results of the
`PaddlePaddle <https://github.com/PaddlePaddle/Paddle>`__ model with the
IR model.
@ -14,11 +16,28 @@ IR model.
Source of the
`model <https://www.paddlepaddle.org.cn/hubdetail?name=mobilenet_v3_large_imagenet_ssld&en_category=ImageClassification>`__.
Preparation
-----------
**Table of contents**:
- `Preparation <#1preparation>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Show Inference on PaddlePaddle Model <#show-inference-on-paddlepaddle-model>`__
- `Convert the Model to OpenVINO IR Format <#convert-the-model-to-openvino-ir-format>`__
- `Select inference device <#select-inference-device>`__
- `Show Inference on OpenVINO Model <#show-inference-on-openvino-model>`__
- `Timing and Comparison <#timing-and-comparison>`__
- `Select inference device <#select-inference-device>`__
- `References <#references>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Imports
~~~~~~~
.. code:: ipython3
@ -61,18 +80,19 @@ Imports
.. parsed-literal::
2023-07-11 22:26:05 INFO: Loading faiss with AVX2 support.
2023-07-11 22:26:05 INFO: Successfully loaded faiss with AVX2 support.
2023-08-15 22:28:07 INFO: Loading faiss with AVX2 support.
2023-08-15 22:28:07 INFO: Successfully loaded faiss with AVX2 support.
Settings
~~~~~~~~
Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Set ``IMAGE_FILENAME`` to the filename of an image to use. Set
``MODEL_NAME`` to the PaddlePaddle model to download from PaddleHub.
``MODEL_NAME`` will also be the base name for the IR model. The notebook
is tested with the
`mobilenet_v3_large_x1_0 <https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/models/Mobile_en.md>`__
`MobileNetV3_large_x1_0 <https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/models/Mobile_en.md>`__
model. Other models may use different preprocessing methods and
therefore require some modification to get the same results on the
original and converted model.
@ -110,8 +130,9 @@ PaddleHub. This may take a while.
Model Extracted to "./model".
Show Inference on PaddlePaddle Model
------------------------------------
Show Inference on PaddlePaddle Model `⇑ <#top>`__
###############################################################################################################################
In the next cell, we load the model, load and display an image, do
inference on that image, and then show the top three prediction results.
@ -130,7 +151,7 @@ inference on that image, and then show the top three prediction results.
.. parsed-literal::
[2023/07/11 22:26:25] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead.
[2023/08/15 22:28:34] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead.
Labrador retriever, 0.75138
German short-haired pointer, 0.02373
Great Dane, 0.01848
@ -182,8 +203,8 @@ the same method.
It is useful to show the output of the ``process_image()`` function, to
see the effect of cropping and resizing. Because of the normalization,
the colors will look strange, and matplotlib will warn about clipping
values.
the colors will look strange, and ``matplotlib`` will warn about
clipping values.
.. code:: ipython3
@ -196,7 +217,7 @@ values.
.. parsed-literal::
2023-07-11 22:26:25 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
2023-08-15 22:28:34 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
.. parsed-literal::
@ -208,7 +229,7 @@ values.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fd87c6ee3a0>
<matplotlib.image.AxesImage at 0x7f28506c8c70>
@ -232,8 +253,9 @@ OpenVINO model.
partition = line.split("\n")[0].partition(" ")
class_id_map[int(partition[0])] = str(partition[-1])
Convert the Model to OpenVINO IR Format
---------------------------------------
Convert the Model to OpenVINO IR Format `⇑ <#top>`__
###############################################################################################################################
Call the OpenVINO Model Optimizer Python API to convert the PaddlePaddle
model to OpenVINO IR, with FP32 precision. ``mo.convert_model`` function
@ -256,10 +278,11 @@ for more information about Model Optimizer.
else:
print(f"{model_xml} already exists.")
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -284,8 +307,9 @@ select device from dropdown list for running inference using OpenVINO
Show Inference on OpenVINO Model
--------------------------------
Show Inference on OpenVINO Model `⇑ <#top>`__
###############################################################################################################################
Load the IR model, get model information, load the image, do inference,
convert the inference to a meaningful result, and show the output. See
@ -332,8 +356,9 @@ information.
.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_23_1.png
Timing and Comparison
---------------------
Timing and Comparison `⇑ <#top>`__
###############################################################################################################################
Measure the time it takes to do inference on fifty images and compare
the result. The timing information gives an indication of performance.
@ -386,7 +411,7 @@ Note that many optimizations are possible to improve the performance.
.. parsed-literal::
PaddlePaddle model on CPU: 0.0074 seconds per image, FPS: 135.48
PaddlePaddle model on CPU: 0.0071 seconds per image, FPS: 141.47
PaddlePaddle result:
Labrador retriever, 0.75138
@ -400,10 +425,11 @@ Note that many optimizations are possible to improve the performance.
.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_27_1.png
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -448,7 +474,7 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0031 seconds per image, FPS: 326.50
OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0030 seconds per image, FPS: 337.97
OpenVINO result:
Labrador retriever, 0.75138
@ -462,8 +488,9 @@ select device from dropdown list for running inference using OpenVINO
.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_30_1.png
References
----------
References `⇑ <#top>`__
###############################################################################################################################
- `PaddleClas <https://github.com/PaddlePaddle/PaddleClas>`__
- `OpenVINO PaddlePaddle

14
docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html
View File

@ -1,11 +1,11 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="103-paddle-to-openvino-classification-with-output_15_3.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 12-Jul-2023 00:11 120883
<a href="103-paddle-to-openvino-classification-with-output_23_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 12-Jul-2023 00:11 224886
<a href="103-paddle-to-openvino-classification-with-output_27_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 12-Jul-2023 00:11 224886
<a href="103-paddle-to-openvino-classification-with-output_30_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 12-Jul-2023 00:11 224886
<a href="103-paddle-to-openvino-classification-with-output_8_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 12-Jul-2023 00:11 224886
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="103-paddle-to-openvino-classification-with-output_15_3.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 16-Aug-2023 01:31 120883
<a href="103-paddle-to-openvino-classification-with-output_23_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 16-Aug-2023 01:31 224886
<a href="103-paddle-to-openvino-classification-with-output_27_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 16-Aug-2023 01:31 224886
<a href="103-paddle-to-openvino-classification-with-output_30_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 16-Aug-2023 01:31 224886
<a href="103-paddle-to-openvino-classification-with-output_8_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 16-Aug-2023 01:31 224886
</pre><hr></body>
</html>

126
docs/notebooks/104-model-tools-with-output.rst
View File

@ -1,37 +1,58 @@
Working with Open Model Zoo Models
==================================
.. _top:
This tutorial shows how to download a model from `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo>`__, convert it
to OpenVINO™ IR format, show information about the model, and benchmark
the model.
the model.
**Table of contents**:
- `OpenVINO and Open Model Zoo Tools <#openvino-and-open-model-zoo-tools>`__
- `Preparation <#preparation>`__
- `Model Name <#model-name>`__
- `Imports <#imports>`__
- `Settings and Configuration <#settings-and-configuration>`__
- `Download a Model from Open Model Zoo <#download-a-model-from-open-model-zoo>`__
- `Convert a Model to OpenVINO IR format <#convert-a-model-to-openvino-ir-format>`__
- `Get Model Information <#get-model-information>`__
- `Run Benchmark Tool <#run-benchmark-tool>`__
- `Benchmark with Different Settings <#benchmark-with-different-settings>`__
OpenVINO and Open Model Zoo Tools `⇑ <#top>`__
###############################################################################################################################
OpenVINO and Open Model Zoo Tools
---------------------------------
OpenVINO and Open Model Zoo tools are listed in the table below.
+------------+--------------+-----------------------------------------+
| Tool | Command | Description |
+============+==============+=========================================+
| Model | omz_download | Download models from Open Model Zoo. |
| Downloader | er | |
| Model | ``omz_downlo | Download models from Open Model Zoo. |
| Downloader | ader`` | |
+------------+--------------+-----------------------------------------+
| Model | omz_converte | Convert Open Model Zoo models to |
| Converter | r | OpenVINO’s IR format. |
| Model | ``omz_conver | Convert Open Model Zoo models to |
| Converter | ter`` | OpenVINO’s IR format. |
+------------+--------------+-----------------------------------------+
| Info | omz_info_dum | Print information about Open Model Zoo |
| Dumper | per | models. |
| Info | ``omz_info_d | Print information about Open Model Zoo |
| Dumper | umper`` | models. |
+------------+--------------+-----------------------------------------+
| Benchmark | benchmark_ap | Benchmark model performance by |
| Tool | p | computing inference time. |
| Benchmark | ``benchmark_ | Benchmark model performance by |
| Tool | app`` | computing inference time. |
+------------+--------------+-----------------------------------------+
Preparation
-----------
Preparation `⇑ <#top>`__
###############################################################################################################################
Model Name `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Model Name
~~~~~~~~~~
Set ``model_name`` to the name of the Open Model Zoo model to use in
this notebook. Refer to the list of
@ -46,8 +67,9 @@ pre-trained models for a full list of models that can be used. Set
# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -61,8 +83,9 @@ Imports
sys.path.append("../utils")
from notebook_utils import DeviceNotFoundAlert, NotebookAlert
Settings and Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~
Settings and Configuration `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Set the file and directory paths. By default, this notebook downloads
models from Open Model Zoo to the ``open_model_zoo_models`` directory in
@ -100,8 +123,9 @@ The following settings can be changed:
base_model_dir: model, omz_cache_dir: cache, gpu_availble: False
Download a Model from Open Model Zoo
------------------------------------
Download a Model from Open Model Zoo `⇑ <#top>`__
###############################################################################################################################
Specify, display and run the Model Downloader command to download the
model.
@ -140,8 +164,9 @@ Downloading mobilenet-v2-pytorch…
Convert a Model to OpenVINO IR format
-------------------------------------
Convert a Model to OpenVINO IR format `⇑ <#top>`__
###############################################################################################################################
Specify, display and run the Model Converter command to convert the
model to OpenVINO IR format. Model conversion may take a while. The
@ -177,25 +202,26 @@ Converting mobilenet-v2-pytorch…
.. parsed-literal::
========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob
ONNX check passed successfully.
========== Converting mobilenet-v2-pytorch to IR (FP16)
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=/tmp/tmpe5yh3lmf --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=/tmp/tmp3q4nxrwu --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument --compress_to_fp16 or set it to false --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/latest/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/latest/openvino_2_0_transition_guide.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /tmp/tmpe5yh3lmf/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /tmp/tmpe5yh3lmf/mobilenet-v2-pytorch.bin
[ SUCCESS ] XML file: /tmp/tmp3q4nxrwu/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /tmp/tmp3q4nxrwu/mobilenet-v2-pytorch.bin
Get Model Information
---------------------
Get Model Information `⇑ <#top>`__
###############################################################################################################################
The Info Dumper prints the following information for Open Model Zoo
models:
@ -240,8 +266,8 @@ information in a dictionary.
'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.\nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.\nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.',
'framework': 'pytorch',
'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE',
'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/accuracy-check.yml',
'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/model.yml',
'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/accuracy-check.yml',
'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/model.yml',
'precisions': ['FP16', 'FP32'],
'quantization_output_precisions': ['FP16-INT8', 'FP32-INT8'],
'subdirectory': 'public/mobilenet-v2-pytorch',
@ -273,8 +299,9 @@ file.
model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True
Run Benchmark Tool
------------------
Run Benchmark Tool `⇑ <#top>`__
###############################################################################################################################
By default, Benchmark Tool runs inference for 60 seconds in asynchronous
mode on CPU. It returns inference speed as latency (milliseconds per
@ -321,7 +348,7 @@ seconds…
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 30.53 ms
[ INFO ] Read model took 29.61 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] data (node: data) : f32 / [N,C,H,W] / [1,3,224,224]
@ -335,7 +362,7 @@ seconds…
[ INFO ] Model outputs:
[ INFO ] prob (node: prob) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 149.89 ms
[ INFO ] Compile model took 154.76 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -357,21 +384,22 @@ seconds…
[ INFO ] Fill input 'data' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 6.66 ms
[ INFO ] First inference took 7.60 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 20136 iterations
[ INFO ] Duration: 15007.99 ms
[ INFO ] Count: 20076 iterations
[ INFO ] Duration: 15004.20 ms
[ INFO ] Latency:
[ INFO ] Median: 4.33 ms
[ INFO ] Average: 4.34 ms
[ INFO ] Min: 3.20 ms
[ INFO ] Max: 11.84 ms
[ INFO ] Throughput: 1341.69 FPS
[ INFO ] Median: 4.34 ms
[ INFO ] Average: 4.35 ms
[ INFO ] Min: 2.53 ms
[ INFO ] Max: 11.71 ms
[ INFO ] Throughput: 1338.03 FPS
Benchmark with Different Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmark with Different Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The ``benchmark_app`` tool displays logging information that is not
always necessary. A more compact result is achieved when the output is
@ -456,9 +484,9 @@ Benchmark command:
command ended
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 327, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 327, in main
benchmark.set_allow_auto_batching(False)
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 63, in set_allow_auto_batching
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 63, in set_allow_auto_batching
self.core.set_property({'ALLOW_AUTO_BATCHING': flag})
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:238:
@ -484,9 +512,9 @@ Benchmark command:
Check 'false' failed at src/plugins/auto/src/plugin_config.cpp:55:
property: ALLOW_AUTO_BATCHING: not supported
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 327, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 327, in main
benchmark.set_allow_auto_batching(False)
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 63, in set_allow_auto_batching
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 63, in set_allow_auto_batching
self.core.set_property({'ALLOW_AUTO_BATCHING': flag})
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:238:
Check 'false' failed at src/plugins/auto/src/plugin_config.cpp:55:

129
docs/notebooks/105-language-quantize-bert-with-output.rst
View File

@ -1,6 +1,8 @@
Quantize NLP models with Post-Training Quantization ​in NNCF
============================================================
.. _top:
This tutorial demonstrates how to apply ``INT8`` quantization to the
Natural Language Processing model known as
`BERT <https://en.wikipedia.org/wiki/BERT_(language_model)>`__, using
@ -22,12 +24,27 @@ and datasets. It consists of the following steps:
- Compare the performance of the original, converted and quantized
models.
**Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Prepare the Model <#prepare-the-model>`__
- `Prepare the Dataset <#prepare-the-dataset>`__
- `Optimize model using NNCF Post-training Quantization API <#optimize-model-using-nncf-post-training-quantization-api>`__
- `Load and Test OpenVINO Model <#load-and-test-openvino-model>`__
- `Select inference device <#select-inference-device>`__
- `Compare F1-score of FP32 and INT8 models <#compare-f1-score-of-fp32-and-int8-models>`__
- `Compare Performance of the Original, Converted and Quantized Models <#compare-performance-of-the-original,-converted-and-quantized-models>`__
.. code:: ipython3
!pip install -q "nncf>=2.5.0" datasets evaluate
Imports
-------
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -56,10 +73,10 @@ Imports
.. parsed-literal::
2023-07-11 22:27:10.887837: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:27:10.921844: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:29:19.942802: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:29:19.975605: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:27:11.494944: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:29:20.517786: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -67,8 +84,9 @@ Imports
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -82,13 +100,15 @@ Settings
os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(MODEL_DIR, exist_ok=True)
Prepare the Model
-----------------
Prepare the Model `⇑ <#top>`__
###############################################################################################################################
Perform the following:
- Download and unpack pre-trained BERT model for MRPC by PyTorch.
- Convert the model to the OpenVINO Intermediate Representation (OpenVINO IR)
- Download and unpack pre-trained BERT model for MRPC by PyTorch.
- Convert the model to the OpenVINO Intermediate Representation
(OpenVINO IR)
.. code:: ipython3
@ -107,12 +127,12 @@ Convert the original PyTorch model to the OpenVINO Intermediate
Representation.
From OpenVINO 2023.0, we can directly convert a model from the PyTorch
format to the OpenVINO IR format using Model Optimizer. Following
format to the OpenVINO IR format using model conversion API. Following
PyTorch model formats are supported:
- torch.nn.Module
- torch.jit.ScriptModule
- torch.jit.ScriptFunction
- ``torch.nn.Module``
- ``torch.jit.ScriptModule``
- ``torch.jit.ScriptFunction``
.. code:: ipython3
@ -142,17 +162,16 @@ PyTorch model formats are supported:
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
warnings.warn("TorchScript will treat type annotations of Tensor "
Prepare the Dataset
-------------------
Prepare the Dataset `⇑ <#top>`__
###############################################################################################################################
We download the `General Language Understanding Evaluation
(GLUE) <https://gluebenchmark.com/>`__ dataset for the MRPC task from
HuggingFace datasets. Then, we tokenize the data with a pre-trained BERT
tokenizer from HuggingFace.
We download the `General Language Understanding Evaluation (GLUE) <https://gluebenchmark.com/>`__ dataset
for the MRPC task from HuggingFace datasets. Then, we tokenize the data
with a pre-trained BERT tokenizer from HuggingFace.
.. code:: ipython3
@ -171,15 +190,9 @@ tokenizer from HuggingFace.
data_source = create_data_source()
Optimize model using NNCF Post-training Quantization API `⇑ <#top>`__
###############################################################################################################################
.. parsed-literal::
Found cached dataset glue (/opt/home/k8sworker/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
Loading cached processed dataset at /opt/home/k8sworker/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-b5f4c739eb2a4a9f.arrow
Optimize model using NNCF Post-training Quantization API
--------------------------------------------------------
`NNCF <https://github.com/openvinotoolkit/nncf>`__ provides a suite of
advanced algorithms for Neural Networks inference optimization in
@ -387,8 +400,8 @@ The optimization process contains the following steps:
.. parsed-literal::
Statistics collection: 100%|██████████| 300/300 [00:24<00:00, 12.02it/s]
Biases correction: 100%|██████████| 74/74 [00:25<00:00, 2.94it/s]
Statistics collection: 100%|██████████| 300/300 [00:24<00:00, 12.04it/s]
Biases correction: 100%|██████████| 74/74 [00:25<00:00, 2.95it/s]
.. code:: ipython3
@ -396,20 +409,22 @@ The optimization process contains the following steps:
compressed_model_xml = Path(MODEL_DIR) / "quantized_bert_mrpc.xml"
ov.serialize(quantized_model, compressed_model_xml)
Load and Test OpenVINO Model
----------------------------
Load and Test OpenVINO Model `⇑ <#top>`__
###############################################################################################################################
To load and test converted model, perform the following:
* Load the model and compile it for selected device.
* Prepare the input.
* Run the inference.
* Get the answer from the model output.
- Load the model and compile it for selected device.
- Prepare the input.
- Run the inference.
- Get the answer from the model output.
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -465,8 +480,9 @@ changing ``sample_idx`` to another value (from 0 to 407).
The same meaning: yes
Compare F1-score of FP32 and INT8 models
----------------------------------------
Compare F1-score of FP32 and INT8 models `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -509,8 +525,8 @@ Compare F1-score of FP32 and INT8 models
F1 score: 0.8995
Compare Performance of the Original, Converted and Quantized Models
-------------------------------------------------------------------
Compare Performance of the Original, Converted and Quantized Models. `⇑ <#top>`__
###############################################################################################################################
Compare the original PyTorch model with OpenVINO converted and quantized
models (``FP32``, ``INT8``) to see the difference in performance. It is
@ -562,14 +578,19 @@ Frames Per Second (FPS) for images.
.. parsed-literal::
PyTorch model on CPU: 0.071 seconds per sentence, SPS: 14.09
IR FP32 model in OpenVINO Runtime/AUTO: 0.022 seconds per sentence, SPS: 45.98
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.010 seconds per sentence, SPS: 98.77
We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
.. parsed-literal::
PyTorch model on CPU: 0.070 seconds per sentence, SPS: 14.22
IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 48.42
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.010 seconds per sentence, SPS: 98.01
Finally, measure the inference performance of OpenVINO ``FP32`` and
``INT8`` models. For this purpose, use `Benchmark
Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
``INT8`` models. For this purpose, use
`Benchmark Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
in OpenVINO.
**Note**: The ``benchmark_app`` tool is able to measure the
@ -600,9 +621,9 @@ in OpenVINO.
[ ERROR ] Check 'false' failed at src/inference/src/core.cpp:84:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 103, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 103, in main
benchmark.print_version_info()
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 48, in print_version_info
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 48, in print_version_info
for device, version in self.core.get_versions(self.device).items():
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:84:
Device with "device" name is not registered in the OpenVINO Runtime
@ -628,9 +649,9 @@ in OpenVINO.
[ ERROR ] Check 'false' failed at src/inference/src/core.cpp:84:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 103, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 103, in main
benchmark.print_version_info()
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 48, in print_version_info
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 48, in print_version_info
for device, version in self.core.get_versions(self.device).items():
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:84:
Device with "device" name is not registered in the OpenVINO Runtime

162
docs/notebooks/106-auto-device-with-output.rst
View File

@ -1,6 +1,8 @@
Automatic Device Selection with OpenVINO™
=========================================
.. _top:
The `Auto
device <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
(or AUTO in short) selects the most suitable device for inference by
@ -25,10 +27,36 @@ immediately on the CPU and then transparently shifts inference to the
GPU, once it is ready. This dramatically reduces the time to execute
first inference.
.. image:: https://camo.githubusercontent.com/cc526c3f5fc992cc7176d097894303248adbd04b4d158bd98e65edc8270af5fc/68747470733a2f2f757365722d696d616765732e67697468756275736572636f6e74656e742e636f6d2f31353730393732332f3136313435313834372d37353965326264622d373062632d343633642d393831382d3430306330636366336331362e706e67
.. figure:: https://user-images.githubusercontent.com/15709723/161451847-759e2bdb-70bc-463d-9818-400c0ccf3c16.png
:alt: auto
auto
**Table of contents**:
- `Import modules and create Core <#import-modules-and-create-core>`__
- `Convert the model to OpenVINO IR format <#convert-the-model-to-openvino-ir-format>`__
- `(1) Simplify selection logic <#1-simplify-selection-logic>`__
- `Default behavior of Core::compile_model API without device_name <#default-behavior-of-core::compile_model-api-without-device_name>`__
- `Explicitly pass AUTO as device_name to Core::compile_model API <#explicitly-pass-auto-as-device_name-to-core::compile_model-api>`__
- `(2) Improve the first inference latency <#2-improve-the-first-inference-latency>`__
- `Load an Image <#load-an-image>`__
- `Load the model to GPU device and perform inference <#load-the-model-to-gpu-device-and-perform-inference>`__
- `Load the model using AUTO device and do inference <#load-the-model-using-auto-device-and-do-inference>`__
- `(3) Achieve different performance for different targets <#3-achieve-different-performance-for-different-targets>`__
- `Class and callback definition <#class-and-callback-definition>`__
- `Inference with THROUGHPUT hint <#inference-with-throughput-hint>`__
- `Inference with LATENCY hint <#inference-with-latency-hint>`__
- `Difference in FPS and latency <#difference-in-fps-and-latency>`__
Import modules and create Core `⇑ <#top>`__
###############################################################################################################################
Import modules and create Core
------------------------------
.. code:: ipython3
@ -50,28 +78,27 @@ Import modules and create Core
device to have meaningful results.
Convert the model to OpenVINO IR format
---------------------------------------
Convert the model to OpenVINO IR format `⇑ <#top>`__
###############################################################################################################################
This tutorial uses
`resnet50 <https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html#resnet50>`__
model from
`torchvision <https://pytorch.org/vision/main/index.html?highlight=torchvision#module-torchvision>`__
library. ResNet 50 is image classification model pre-trained on ImageNet
dataset described in paper `“Deep Residual Learning for Image
Recognition” <https://arxiv.org/abs/1512.03385>`__. From OpenVINO
dataset described in paper `“Deep Residual Learning for Image Recognition” <https://arxiv.org/abs/1512.03385>`__. From OpenVINO
2023.0, we can directly convert a model from the PyTorch format to the
OpenVINO IR format using Model Optimizer. To convert model, we should
provide model object instance into ``mo.convert_model`` function,
OpenVINO IR format using model conversion API. To convert model, we
should provide model object instance into ``mo.convert_model`` function,
optionally, we can specify input shape for conversion (by default models
from PyTorch converted with dynamic input shapes). ``mo.convert_model``
returns openvino.runtime.Model object ready to be loaded on device with
``openvino.runtime.Core().compile_model`` or serialized for next usage
with ``openvino.runtime.serialize``.
returns openvino.runtime.Model object ready to be loaded on a device
with ``openvino.runtime.Core().compile_model`` or serialized for next
usage with ``openvino.runtime.serialize``.
For more information about Model Optimizer, see the `Model Optimizer
Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
For more information about model conversion API, see this
`page <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
.. code:: ipython3
@ -99,14 +126,15 @@ Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_O
IR model saved to model/resnet50.xml
(1) Simplify selection logic
----------------------------
(1) Simplify selection logic `⇑ <#top>`__
###############################################################################################################################
Default behavior of Core::compile_model API without device_name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Default behavior of Core::compile_model API without device_name `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
By default, ``compile_model`` API will select **AUTO** as
``device_name`` if no device is specified.
By default, ``compile_model`` API will select **AUTO** as ``device_name`` if no
device is specified.
.. code:: ipython3
@ -137,11 +165,11 @@ By default, ``compile_model`` API will select **AUTO** as
Deleted compiled_model
Explicitly pass AUTO as device_name to Core::compile_model API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Explicitly pass AUTO as device_name to Core::compile_model API `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
It is optional, but passing AUTO explicitly as ``device_name`` may
improve readability of your code.
It is optional, but passing AUTO explicitly as
``device_name`` may improve readability of your code.
.. code:: ipython3
@ -171,13 +199,13 @@ improve readability of your code.
Deleted compiled_model
(2) Improve the first inference latency
---------------------------------------
(2) Improve the first inference latency `⇑ <#top>`__
###############################################################################################################################
One of the benefits of using AUTO device selection is reducing FIL
(first inference latency). FIL is the model compilation time combined
with the first inference execution time. Using the CPU device explicitly
will produce the shortest first inference latency, as the OpenVINO graph
One of the benefits of using AUTO device selection is reducing FIL (first inference
latency). FIL is the model compilation time combined with the first
inference execution time. Using the CPU device explicitly will produce
the shortest first inference latency, as the OpenVINO graph
representation loads quickly on CPU, using just-in-time (JIT)
compilation. The challenge is with GPU devices since OpenCL graph
complication to GPU-optimized kernels takes a few seconds to complete.
@ -185,11 +213,12 @@ This initialization time may be intolerable for some applications. To
avoid this delay, the AUTO uses CPU transparently as the first inference
device until GPU is ready.
Load an Image
~~~~~~~~~~~~~
Load an Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
torchvision library provides model specific input transformation
function, we will reuse it for preparing input data.
Torchvision library provides model specific
input transformation function, we will reuse it for preparing input
data.
.. code:: ipython3
@ -209,8 +238,9 @@ function, we will reuse it for preparing input data.
Load the model to GPU device and perform inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the model to GPU device and perform inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -236,8 +266,8 @@ Load the model to GPU device and perform inference
A GPU device is not available. Available devices are: ['CPU']
Load the model using AUTO device and do inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the model using AUTO device and do inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When GPU is the best available device, the first few inferences will be
executed on CPU until GPU is ready.
@ -268,8 +298,8 @@ executed on CPU until GPU is ready.
# Deleted model will wait for compiling on the selected device to complete.
del compiled_model
(3) Achieve different performance for different targets
-------------------------------------------------------
(3) Achieve different performance for different targets `⇑ <#top>`__
###############################################################################################################################
It is an advantage to define **performance hints** when using Automatic
Device Selection. By specifying a **THROUGHPUT** or **LATENCY** hint,
@ -280,14 +310,13 @@ hints do not require any device-specific settings and they are
completely portable between devices – meaning AUTO can configure the
performance hint on whichever device is being used.
For more information, refer to the `Performance
Hints <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html#performance-hints>`__
section of `Automatic Device
Selection <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
For more information, refer to the `Performance Hints <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html#performance-hints>`__
section of `Automatic Device Selection <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
article.
Class and callback definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Class and callback definition `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -385,8 +414,9 @@ Class and callback definition
metrics_update_interval = 10
metrics_update_num = 6
Inference with THROUGHPUT hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inference with THROUGHPUT hint `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Loop for inference and update the FPS/Latency every
@metrics_update_interval seconds.
@ -424,17 +454,18 @@ Loop for inference and update the FPS/Latency every
Compiling Model for AUTO device with THROUGHPUT hint
Start inference, 6 groups of FPS/latency will be measured over 10s intervals
throughput: 190.70fps, latency: 29.76ms, time interval: 10.02s
throughput: 191.95fps, latency: 30.48ms, time interval: 10.00s
throughput: 192.78fps, latency: 30.40ms, time interval: 10.00s
throughput: 191.39fps, latency: 30.62ms, time interval: 10.00s
throughput: 192.18fps, latency: 30.44ms, time interval: 10.03s
throughput: 191.33fps, latency: 30.62ms, time interval: 10.00s
throughput: 189.24fps, latency: 30.04ms, time interval: 10.00s
throughput: 192.12fps, latency: 30.48ms, time interval: 10.01s
throughput: 191.27fps, latency: 30.64ms, time interval: 10.00s
throughput: 190.87fps, latency: 30.69ms, time interval: 10.01s
throughput: 189.50fps, latency: 30.89ms, time interval: 10.02s
throughput: 190.30fps, latency: 30.79ms, time interval: 10.01s
Done
Inference with LATENCY hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inference with LATENCY hint `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Loop for inference and update the FPS/Latency for each
@metrics_update_interval seconds
@ -473,17 +504,18 @@ Loop for inference and update the FPS/Latency for each
Compiling Model for AUTO Device with LATENCY hint
Start inference, 6 groups fps/latency will be out with 10s interval
throughput: 136.99fps, latency: 6.75ms, time interval: 10.00s
throughput: 140.91fps, latency: 6.74ms, time interval: 10.01s
throughput: 140.83fps, latency: 6.74ms, time interval: 10.00s
throughput: 140.90fps, latency: 6.74ms, time interval: 10.00s
throughput: 140.83fps, latency: 6.74ms, time interval: 10.00s
throughput: 140.85fps, latency: 6.74ms, time interval: 10.00s
throughput: 138.76fps, latency: 6.68ms, time interval: 10.00s
throughput: 141.79fps, latency: 6.70ms, time interval: 10.00s
throughput: 142.39fps, latency: 6.68ms, time interval: 10.00s
throughput: 142.30fps, latency: 6.68ms, time interval: 10.00s
throughput: 142.30fps, latency: 6.68ms, time interval: 10.01s
throughput: 142.53fps, latency: 6.67ms, time interval: 10.00s
Done
Difference in FPS and latency
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Difference in FPS and latency `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3

4
docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_25_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:03224259b8d1a7dd982c9d5b2baae8bdca495cbe008f83bc5b0a0fd3ecc47ab6
size 27172
oid sha256:7858463aadc74dfe6af6906da15e208305234a13a8463c4f4e4632630fffde70
size 27107

4
docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_26_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d962a84b77491e4c1f9bdcb4e0c58f6aae55f11f8e59ae747077b166e706fd76
size 40994
oid sha256:48a1f70dac1b326af8f205247fc377c9fb72286526c2d19507ada4b520a779ed
size 39987

12
docs/notebooks/106-auto-device-with-output_files/index.html
View File

@ -1,10 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/106-auto-device-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/106-auto-device-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/106-auto-device-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="106-auto-device-with-output_12_0.jpg">106-auto-device-with-output_12_0.jpg</a> 12-Jul-2023 00:11 121563
<a href="106-auto-device-with-output_12_0.png">106-auto-device-with-output_12_0.png</a> 12-Jul-2023 00:11 869661
<a href="106-auto-device-with-output_25_0.png">106-auto-device-with-output_25_0.png</a> 12-Jul-2023 00:11 27172
<a href="106-auto-device-with-output_26_0.png">106-auto-device-with-output_26_0.png</a> 12-Jul-2023 00:11 40994
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/106-auto-device-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="106-auto-device-with-output_12_0.jpg">106-auto-device-with-output_12_0.jpg</a> 16-Aug-2023 01:31 121563
<a href="106-auto-device-with-output_12_0.png">106-auto-device-with-output_12_0.png</a> 16-Aug-2023 01:31 869661
<a href="106-auto-device-with-output_25_0.png">106-auto-device-with-output_25_0.png</a> 16-Aug-2023 01:31 27107
<a href="106-auto-device-with-output_26_0.png">106-auto-device-with-output_26_0.png</a> 16-Aug-2023 01:31 39987
</pre><hr></body>
</html>

78
docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst
View File

@ -1,6 +1,8 @@
Quantize Speech Recognition Models using NNCF PTQ API
=====================================================
.. _top:
This tutorial demonstrates how to use the NNCF (Neural Network
Compression Framework) 8-bit quantization in post-training mode (without
the fine-tuning pipeline) to optimize the speech recognition model,
@ -19,8 +21,24 @@ steps:
- Compare performance of the original and quantized models.
- Compare Accuracy of the Original and Quantized Models.
Download and prepare model
--------------------------
**Table of contents**:
- `Download and prepare model <#download-and-prepare-model>`__
- `Obtain Pytorch model representation <#obtain-pytorch-model-representation>`__
- `Convert model to OpenVINO Intermediate Representation <#convert-model-to-openvino-intermediate-representation>`__
- `Prepare inference data <#prepare-inference-data>`__
- `Check model inference result <#check-model-inference-result>`__
- `Validate model accuracy on dataset <#validate-model-accuracy-on-dataset>`__
- `Quantization <#quantization>`__
- `Check INT8 model inference result <#check-int8-model-inference-result>`__
- `Compare Performance of the Original and Quantized Models <#compare-performance-of-the-original-and-quantized-models>`__
- `Compare Accuracy of the Original and Quantized Models <#compare-accuracy-of-the-original-and-quantized-models>`__
Download and prepare model `⇑ <#top>`__
###############################################################################################################################
data2vec is a framework for self-supervised representation learning for
images, speech, and text as described in `data2vec: A General Framework
@ -38,8 +56,9 @@ In our case, we will use ``data2vec-audio-base-960h`` model, which was
finetuned on 960 hours of audio from LibriSpeech Automatic Speech
Recognition corpus and distributed as part of HuggingFace transformers.
Obtain Pytorch model representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Obtain Pytorch model representation `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For instantiating PyTorch model class, we should use
``Data2VecAudioForCTC.from_pretrained`` method with providing model ID
@ -53,7 +72,7 @@ model specific pre- and post-processing steps.
.. code:: ipython3
!pip install -q 'openvino-dev>=2023.0.0' 'nncf>=2.5.0'
!pip install -q "openvino-dev>=2023.0.0" "nncf>=2.5.0"
!pip install -q soundfile librosa transformers onnx
.. code:: ipython3
@ -63,8 +82,9 @@ model specific pre- and post-processing steps.
processor = Wav2Vec2Processor.from_pretrained("facebook/data2vec-audio-base-960h")
model = Data2VecAudioForCTC.from_pretrained("facebook/data2vec-audio-base-960h")
Convert model to OpenVINO Intermediate Representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Convert model to OpenVINO Intermediate Representation `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -134,11 +154,12 @@ Convert model to OpenVINO Intermediate Representation
Read IR model from model/data2vec-audo-base.xml
Prepare inference data
~~~~~~~~~~~~~~~~~~~~~~
Prepare inference data `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For demonstration purposes, we will use short dummy version of
librispeach dataset - ``patrickvonplaten/librispeech_asr_dummy`` to
LibriSpeech dataset - ``patrickvonplaten/librispeech_asr_dummy`` to
speed up model evaluation. Model accuracy can be different from reported
in the paper. For reproducing original accuracy, use ``librispeech_asr``
dataset.
@ -174,8 +195,9 @@ dataset.
Loading cached processed dataset at /home/adrian/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc/cache-4e0f4916cd205b24.arrow
Check model inference result
----------------------------
Check model inference result `⇑ <#top>`__
###############################################################################################################################
The code below is used for running model inference on a single sample
from the dataset. It contains the following steps:
@ -247,8 +269,9 @@ For reference, see the same function provided for OpenVINO model.
Validate model accuracy on dataset
----------------------------------
Validate model accuracy on dataset `⇑ <#top>`__
###############################################################################################################################
For model accuracy evaluation, `Word Error
Rate <https://en.wikipedia.org/wiki/Word_error_rate>`__ metric can be
@ -307,8 +330,9 @@ library.
[OpenVino] Word Error Rate: 0.0383
Quantization
------------
Quantization `⇑ <#top>`__
###############################################################################################################################
`NNCF <https://github.com/openvinotoolkit/nncf>`__ provides a suite of
advanced algorithms for Neural Networks inference optimization in
@ -318,9 +342,11 @@ Create a quantized model from the pre-trained ``FP16`` model and the
calibration dataset. The optimization process contains the following
steps:
1. Create a Dataset for quantization.
2. Run ``nncf.quantize`` for getting an optimized model. The ``nncf.quantize`` function provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using the ``ignored_scope`` parameter. ``advanced_parameters`` can be used to specify advanced quantization parameters for fine-tuning the quantization algorithm. In this tutorial we pass range estimator parameters for activations. For more information, see `Tune quantization parameters <https://docs.openvino.ai/2023.0/basic_quantization_flow.html#tune-quantization-parameters>`__.
3. Serialize OpenVINO IR model using ``openvino.runtime.serialize`` function.
::
1. Create a Dataset for quantization.
2. Run `nncf.quantize` for getting an optimized model. The `nncf.quantize` function provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using the `ignored_scope` parameter. `advanced_parameters` can be used to specify advanced quantization parameters for fine-tuning the quantization algorithm. In this tutorial we pass range estimator parameters for activations. For more information see [Tune quantization parameters](https://docs.openvino.ai/2023.0/basic_quantization_flow.html#tune-quantization-parameters).
3. Serialize OpenVINO IR model using `openvino.runtime.serialize` function.
.. code:: ipython3
@ -589,8 +615,9 @@ saved using ``serialize`` function.
quantized_model_path = Path(f"{MODEL_NAME}_openvino_model/{MODEL_NAME}_quantized.xml")
serialize(quantized_model, str(quantized_model_path))
Check INT8 model inference result
---------------------------------
Check INT8 model inference result `⇑ <#top>`__
###############################################################################################################################
``INT8`` model is the same in usage like the original one. We need to
read it, using the ``core.read_model`` method and load on the device,
@ -628,8 +655,8 @@ using ``core.compile_model``. After that, we can reuse the same
Compare Performance of the Original and Quantized Models
--------------------------------------------------------
Compare Performance of the Original and Quantized Models `⇑ <#top>`__
###############################################################################################################################
`Benchmark
Tool <https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html>`__
@ -791,8 +818,9 @@ is used to measure the inference performance of the ``FP16`` and
[ INFO ] Throughput: 38.24 FPS
Compare Accuracy of the Original and Quantized Models
-----------------------------------------------------
Compare Accuracy of the Original and Quantized Models `⇑ <#top>`__
###############################################################################################################################
Finally, calculate WER metric for the ``INT8`` model representation and
compare it with the ``FP16`` result.

1375
docs/notebooks/108-gpu-device-with-output.rst Normal file
View File

File diff suppressed because it is too large Load Diff

145
docs/notebooks/109-latency-tricks-with-output.rst
View File

@ -1,6 +1,8 @@
Performance tricks in OpenVINO for latency mode
===============================================
.. _top:
The goal of this notebook is to provide a step-by-step tutorial for
improving performance for inferencing in a latency mode. Low latency is
especially desired in real-time applications when the results are needed
@ -11,8 +13,8 @@ simulate a camera application that provides frames one by one.
The performance tips applied in this notebook could be summarized in the
following figure. Some of the steps below can be applied to any device
at any stage, e.g., “shared_memory”; some can be used only to specific
devices, e.g., “inference num threads” to CPU. As the number of
at any stage, e.g., ``shared_memory``; some can be used only to specific
devices, e.g., ``INFERENCE_NUM_THREADS`` to CPU. As the number of
potential configurations is vast, we recommend looking at the steps
below and then apply a trial-and-error approach. You can incorporate
many hints simultaneously, like more inference threads + shared memory.
@ -38,6 +40,33 @@ optimize performance on OpenVINO IR files in
run this notebook on your computer with your model to learn which of
them makes sense in your case.
All the following tricks were run with OpenVINO 2022.3. Future
versions of OpenVINO may include various optimizations that may
result in different performance.
A similar notebook focused on the throughput mode is available
`here <109-throughput-tricks.ipynb>`__.
**Table of contents**:
- `Data <#data>`__
- `Model <#model>`__
- `Hardware <#hardware>`__
- `Helper functions <#helper-functions>`__
- `Optimizations <#optimizations>`__
- `PyTorch model <#pytorch-model>`__
- `ONNX model <#onnx-model>`__
- `OpenVINO IR model <#openvino-ir-model>`__
- `OpenVINO IR model on GPU <#openvino-ir-model-on-gpu>`__
- `OpenVINO IR model + more inference threads <#openvino-ir-model-+-more-inference-threads>`__
- `OpenVINO IR model in latency mode <#openvino-ir-model-in-latency-mode>`__
- `OpenVINO IR model in latency mode + shared memory <#openvino-ir-model-in-latency-mode-+-shared-memory>`__
- `Other tricks <#other-tricks>`__
- `Performance comparison <#performance-comparison>`__
- `Conclusions <#conclusions>`__
Prerequisites
-------------
@ -58,8 +87,9 @@ Prerequisites
sys.path.append("../utils")
import notebook_utils as utils
Data
----
Data `⇑ <#top>`__
###############################################################################################################################
We will use the same image of the dog sitting on a bicycle for all
experiments below. The image is resized and preprocessed to fulfill the
@ -94,12 +124,13 @@ requirements of this particular object detection model.
.. parsed-literal::
<DisplayHandle display_id=5ab5e600f8b1dde3f55ff40ec675d04c>
<DisplayHandle display_id=c492bd5a5310fae2dfdea013f2ab99ab>
Model
-----
Model `⇑ <#top>`__
###############################################################################################################################
We decided to go with
`YOLOv5n <https://github.com/ultralytics/yolov5>`__, one of the
@ -155,8 +186,9 @@ PyTorch Hub and small enough to see the difference in performance.
Adding AutoShape...
Hardware
--------
Hardware `⇑ <#top>`__
###############################################################################################################################
The code below lists the available hardware we will use in the
benchmarking process.
@ -182,8 +214,9 @@ benchmarking process.
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
Helper functions
----------------
Helper functions `⇑ <#top>`__
###############################################################################################################################
We’re defining a benchmark model function to use for all optimized
models below. It runs inference 1000 times, averages the latency time,
@ -317,15 +350,17 @@ the image.
utils.show_array(output_img)
Optimizations
-------------
Optimizations `⇑ <#top>`__
###############################################################################################################################
Below, we present the performance tricks for faster inference in the
latency mode. We release resources after every benchmarking to be sure
the same amount of resource is available for every experiment.
PyTorch model
~~~~~~~~~~~~~
PyTorch model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
First, we’re benchmarking the original PyTorch model without any
optimizations applied. We will treat it as our baseline.
@ -346,12 +381,13 @@ optimizations applied. We will treat it as our baseline.
.. parsed-literal::
PyTorch model on CPU. First inference time: 0.0286 seconds
PyTorch model on CPU: 0.0202 seconds per image (49.58 FPS)
PyTorch model on CPU. First inference time: 0.0227 seconds
PyTorch model on CPU: 0.0191 seconds per image (52.34 FPS)
ONNX model
~~~~~~~~~~
ONNX model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The first optimization is exporting the PyTorch model to ONNX and
running it in OpenVINO. It’s possible, thanks to the ONNX frontend. It
@ -395,18 +431,19 @@ Representation (IR) to leverage the OpenVINO Runtime.
.. parsed-literal::
ONNX model on CPU. First inference time: 0.0173 seconds
ONNX model on CPU: 0.0133 seconds per image (75.16 FPS)
ONNX model on CPU. First inference time: 0.0182 seconds
ONNX model on CPU: 0.0135 seconds per image (74.31 FPS)
OpenVINO IR model
~~~~~~~~~~~~~~~~~
OpenVINO IR model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Let’s convert the ONNX model to OpenVINO Intermediate Representation
(IR) FP16 and run it. Reducing the precision is one of the well-known
methods for faster inference provided the hardware that supports lower
precision, such as FP16 or even INT8. If the hardware doesn’t support
lower precisions, the model will be inferred in FP32 automatically. We
lower precision, the model will be inferred in FP32 automatically. We
could also use quantization (INT8), but we should experience a little
accuracy drop. That’s why we skip that step in this notebook.
@ -433,12 +470,13 @@ accuracy drop. That’s why we skip that step in this notebook.
.. parsed-literal::
OpenVINO model on CPU. First inference time: 0.0163 seconds
OpenVINO model on CPU: 0.0132 seconds per image (75.64 FPS)
OpenVINO model on CPU. First inference time: 0.0157 seconds
OpenVINO model on CPU: 0.0134 seconds per image (74.40 FPS)
OpenVINO IR model on GPU
~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO IR model on GPU `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Usually, a GPU device is faster than a CPU, so let’s run the above model
on the GPU. Please note you need to have an Intel GPU and `install
@ -461,8 +499,9 @@ execution.
del ov_gpu_model # release resources
OpenVINO IR model + more inference threads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO IR model + more inference threads `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
There is a possibility to add a config for any device (CPU in this
case). We will increase the number of threads to an equal number of our
@ -491,11 +530,12 @@ our case.
.. parsed-literal::
OpenVINO model + more threads on CPU. First inference time: 0.0151 seconds
OpenVINO model + more threads on CPU: 0.0132 seconds per image (75.68 FPS)
OpenVINO model + more threads on CPU: 0.0134 seconds per image (74.36 FPS)
OpenVINO IR model in latency mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO IR model in latency mode `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO offers a virtual device called
`AUTO <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__,
@ -520,12 +560,13 @@ devices as well.
.. parsed-literal::
OpenVINO model on AUTO. First inference time: 0.0156 seconds
OpenVINO model on AUTO: 0.0135 seconds per image (73.93 FPS)
OpenVINO model on AUTO. First inference time: 0.0159 seconds
OpenVINO model on AUTO: 0.0136 seconds per image (73.57 FPS)
OpenVINO IR model in latency mode + shared memory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO IR model in latency mode + shared memory `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO is a C++ toolkit with Python wrappers (API). The default
behavior in the Python API is copying the input to the additional buffer
@ -554,21 +595,24 @@ performance!
.. parsed-literal::
OpenVINO model + shared memory on AUTO. First inference time: 0.0144 seconds
OpenVINO model + shared memory on AUTO: 0.0053 seconds per image (187.92 FPS)
OpenVINO model + shared memory on AUTO. First inference time: 0.0139 seconds
OpenVINO model + shared memory on AUTO: 0.0054 seconds per image (184.61 FPS)
Other tricks
~~~~~~~~~~~~
Other tricks `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
There are other tricks for performance improvement, especially
quantization and prepostprocessing. To get even more from your model,
please visit
`111-detection-quantization <../111-detection-quantization>`__ and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
Performance comparison
----------------------
There are other tricks for performance improvement, such as quantization
and pre-post-processing or dedicated to throughput mode. To get even
more from your model, please visit
`111-detection-quantization <../111-detection-quantization>`__,
`118-optimize-preprocessing <../118-optimize-preprocessing>`__, and
`109-throughput-tricks <109-throughput-tricks.ipynb>`__.
Performance comparison `⇑ <#top>`__
###############################################################################################################################
The following graphical comparison is valid for the selected model and
hardware simultaneously. If you cannot see any improvement between some
@ -604,8 +648,9 @@ steps, just skip them.
.. image:: 109-latency-tricks-with-output_files/109-latency-tricks-with-output_30_0.png
Conclusions
-----------
Conclusions `⇑ <#top>`__
###############################################################################################################################
We already showed the steps needed to improve the performance of an
object detection model. Even if you experience much better performance

4
docs/notebooks/109-latency-tricks-with-output_files/109-latency-tricks-with-output_30_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3683f4df707fff21d7f1e5329acbdfb9d00031ded856582d893c05a67f726d4e
size 57103
oid sha256:9d03578132292067edf38c21858cb3b2ed61780d3ff47f1962330823f4abee26
size 56954

20
docs/notebooks/109-latency-tricks-with-output_files/index.html
View File

@ -1,14 +1,14 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/109-latency-tricks-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/109-latency-tricks-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/109-latency-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-latency-tricks-with-output_14_0.jpg">109-latency-tricks-with-output_14_0.jpg</a> 12-Jul-2023 00:11 162715
<a href="109-latency-tricks-with-output_17_0.jpg">109-latency-tricks-with-output_17_0.jpg</a> 12-Jul-2023 00:11 162715
<a href="109-latency-tricks-with-output_19_0.jpg">109-latency-tricks-with-output_19_0.jpg</a> 12-Jul-2023 00:11 162756
<a href="109-latency-tricks-with-output_23_0.jpg">109-latency-tricks-with-output_23_0.jpg</a> 12-Jul-2023 00:11 162756
<a href="109-latency-tricks-with-output_25_0.jpg">109-latency-tricks-with-output_25_0.jpg</a> 12-Jul-2023 00:11 162756
<a href="109-latency-tricks-with-output_27_0.jpg">109-latency-tricks-with-output_27_0.jpg</a> 12-Jul-2023 00:11 162756
<a href="109-latency-tricks-with-output_30_0.png">109-latency-tricks-with-output_30_0.png</a> 12-Jul-2023 00:11 57103
<a href="109-latency-tricks-with-output_4_0.jpg">109-latency-tricks-with-output_4_0.jpg</a> 12-Jul-2023 00:11 155828
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/109-latency-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-latency-tricks-with-output_14_0.jpg">109-latency-tricks-with-output_14_0.jpg</a> 16-Aug-2023 01:31 162715
<a href="109-latency-tricks-with-output_17_0.jpg">109-latency-tricks-with-output_17_0.jpg</a> 16-Aug-2023 01:31 162715
<a href="109-latency-tricks-with-output_19_0.jpg">109-latency-tricks-with-output_19_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-latency-tricks-with-output_23_0.jpg">109-latency-tricks-with-output_23_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-latency-tricks-with-output_25_0.jpg">109-latency-tricks-with-output_25_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-latency-tricks-with-output_27_0.jpg">109-latency-tricks-with-output_27_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-latency-tricks-with-output_30_0.png">109-latency-tricks-with-output_30_0.png</a> 16-Aug-2023 01:31 56954
<a href="109-latency-tricks-with-output_4_0.jpg">109-latency-tricks-with-output_4_0.jpg</a> 16-Aug-2023 01:31 155828
</pre><hr></body>
</html>

717
docs/notebooks/109-throughput-tricks-with-output.rst Normal file
View File

@ -0,0 +1,717 @@
Performance tricks in OpenVINO for throughput mode
==================================================
.. _top:
The goal of this notebook is to provide a step-by-step tutorial for
improving performance for inferencing in a throughput mode. High
throughput is especially desired in applications when the results are
not expected to appear as soon as possible but to lower the whole
processing time. This notebook assumes computer vision workflow and uses
`YOLOv5n <https://github.com/ultralytics/yolov5>`__ model. We will
simulate a video processing application that has access to all frames at
once (e.g. video editing).
The performance tips applied in this notebook could be summarized in the
following figure. Some of the steps below can be applied to any device
at any stage, e.g., batch size; some can be used only to specific
devices, e.g., inference threads number to CPU. As the number of
potential configurations is vast, we recommend looking at the steps
below and then apply a trial-and-error approach. You can incorporate
many hints simultaneously, like more inference threads + async
processing. It should give even better performance, but we recommend
testing it anyway.
The quantization and pre-post-processing API are not included here as
they change the precision (quantization) or processing graph
(prepostprocessor). You can find examples of how to apply them to
optimize performance on OpenVINO IR files in
`111-detection-quantization <../111-detection-quantization>`__ and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
|image0|
**NOTE**: Many of the steps presented below will give you better
performance. However, some of them may not change anything if they
are strongly dependent on either the hardware or the model. Please
run this notebook on your computer with your model to learn which of
them makes sense in your case.
All the following tricks were run with OpenVINO 2022.3. Future
versions of OpenVINO may include various optimizations that may
result in different performance.
A similar notebook focused on the latency mode is available
`here <109-latency-tricks.ipynb>`__.
**Table of contents**:
- `Data <#data>`__
- `Model <#model>`__
- `Hardware <#hardware>`__
- `Helper functions <#helper-functions>`__
- `Optimizations <#optimizations>`__
- `PyTorch model <#pytorch-model>`__
- `OpenVINO IR model <#openvino-ir-model>`__
- `OpenVINO IR model + bigger batch <#openvino-ir-model-+-bigger-batch>`__
- `OpenVINO IR model in throughput mode <#openvino-ir-model-in-throughput-mode>`__
- `OpenVINO IR model in throughput mode on GPU <#openvino-ir-model-in-throughput-mode-on-gpu>`__
- `OpenVINO IR model in throughput mode on AUTO <#openvino-ir-model-in-throughput-mode-on-auto>`__
- `OpenVINO IR model in cumulative throughput mode on AUTO <#openvino-ir-model-in-cumulative-throughput-mode-on-auto>`__
- `OpenVINO IR model in cumulative throughput mode on AUTO + asynchronous processing <#openvino-ir-model-in-cumulative-throughput-mode-on-auto-+-asynchronous-processing>`__
- `Other tricks <#other-tricks>`__
- `Performance comparison <#performance-comparison>`__
- `Conclusions <#conclusions>`__
Prerequisites
-------------
.. |image0| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/4547501/e1a6e230-7c80-491a-8732-02515c556f1b
.. code:: ipython3
!pip install -q seaborn ultralytics
.. code:: ipython3
import sys
import time
from pathlib import Path
from typing import Any, List, Tuple
sys.path.append("../utils")
import notebook_utils as utils
Data `⇑ <#top>`__
###############################################################################################################################
We will use the same image of the dog sitting on a bicycle copied 1000
times to simulate the video with 1000 frames (about 33s). The image is
resized and preprocessed to fulfill the requirements of this particular
object detection model.
.. code:: ipython3
import numpy as np
import cv2
FRAMES_NUMBER = 1024
IMAGE_WIDTH = 640
IMAGE_HEIGHT = 480
# load image
image = utils.load_image("../data/image/coco_bike.jpg")
image = cv2.resize(image, dsize=(IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_AREA)
# preprocess it for YOLOv5
input_image = image / 255.0
input_image = np.transpose(input_image, axes=(2, 0, 1))
input_image = np.expand_dims(input_image, axis=0)
# simulate video with many frames
video_frames = np.tile(input_image, (FRAMES_NUMBER, 1, 1, 1, 1))
# show the image
utils.show_array(image)
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_4_0.jpg
.. parsed-literal::
<DisplayHandle display_id=5216512dd310a9e1c0f8296b3aca9633>
Model `⇑ <#top>`__
###############################################################################################################################
We decided to go with
`YOLOv5n <https://github.com/ultralytics/yolov5>`__, one of the
state-of-the-art object detection models, easily available through the
PyTorch Hub and small enough to see the difference in performance.
.. code:: ipython3
import torch
from IPython.utils import io
# directory for all models
base_model_dir = Path("model")
model_name = "yolov5n"
model_path = base_model_dir / model_name
# load YOLOv5n from PyTorch Hub
pytorch_model = torch.hub.load("ultralytics/yolov5", "custom", path=model_path, device="cpu", skip_validation=True)
# don't print full model architecture
with io.capture_output():
pytorch_model.eval()
.. parsed-literal::
Using cache found in /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master
YOLOv5 🚀 2023-4-21 Python-3.8.10 torch-1.13.1+cpu CPU
Fusing layers...
YOLOv5n summary: 213 layers, 1867405 parameters, 0 gradients
Adding AutoShape...
.. parsed-literal::
requirements: /opt/home/k8sworker/.cache/torch/hub/requirements.txt not found, check failed.
Hardware `⇑ <#top>`__
###############################################################################################################################
The code below lists the available hardware we will use in the
benchmarking process.
**NOTE**: The hardware you have is probably completely different from
ours. It means you can see completely different results.
.. code:: ipython3
import openvino.runtime as ov
# initialize OpenVINO
core = ov.Core()
# print available devices
for device in core.available_devices:
device_name = core.get_property(device, "FULL_DEVICE_NAME")
print(f"{device}: {device_name}")
.. parsed-literal::
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
Helper functions `⇑ <#top>`__
###############################################################################################################################
We’re defining a benchmark model function to use for all optimizations
below. It runs inference for 1000 frames and prints average frames per
second (FPS).
.. code:: ipython3
from openvino.runtime import AsyncInferQueue
def benchmark_model(model: Any, frames: np.ndarray, async_queue: AsyncInferQueue = None, benchmark_name: str = "OpenVINO model", device_name: str = "CPU") -> float:
"""
Helper function for benchmarking the model. It measures the time and prints results.
"""
# measure the first inference separately - it may be slower as it contains also initialization
start = time.perf_counter()
model(frames[0])
if async_queue:
async_queue.wait_all()
end = time.perf_counter()
first_infer_time = end - start
print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")
# benchmarking
start = time.perf_counter()
for batch in frames:
model(batch)
# wait for all threads if async processing
if async_queue:
async_queue.wait_all()
end = time.perf_counter()
# elapsed time
infer_time = end - start
# print second per image and FPS
mean_infer_time = infer_time / FRAMES_NUMBER
mean_fps = FRAMES_NUMBER / infer_time
print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")
return mean_fps
The following functions aim to post-process results and draw boxes on
the image.
.. code:: ipython3
# https://gist.github.com/AruniRC/7b3dadd004da04c80198557db5da4bda
classes = [
"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant",
"stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra",
"giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite",
"baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork",
"knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut",
"cake", "chair", "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard",
"cell phone", "microwave", "oven", "oaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear",
"hair drier", "toothbrush"
]
# Colors for the classes above (Rainbow Color Map).
colors = cv2.applyColorMap(
src=np.arange(0, 255, 255 / len(classes), dtype=np.float32).astype(np.uint8),
colormap=cv2.COLORMAP_RAINBOW,
).squeeze()
def postprocess(detections: np.ndarray) -> List[Tuple]:
"""
Postprocess the raw results from the model.
"""
# candidates - probability > 0.25
detections = detections[detections[..., 4] > 0.25]
boxes = []
labels = []
scores = []
for obj in detections:
xmin, ymin, ww, hh = obj[:4]
score = obj[4]
label = np.argmax(obj[5:])
# Create a box with pixels coordinates from the box with normalized coordinates [0,1].
boxes.append(
tuple(map(int, (xmin - ww // 2, ymin - hh // 2, ww, hh)))
)
labels.append(int(label))
scores.append(float(score))
# Apply non-maximum suppression to get rid of many overlapping entities.
# See https://paperswithcode.com/method/non-maximum-suppression
# This algorithm returns indices of objects to keep.
indices = cv2.dnn.NMSBoxes(
bboxes=boxes, scores=scores, score_threshold=0.25, nms_threshold=0.5
)
# If there are no boxes.
if len(indices) == 0:
return []
# Filter detected objects.
return [(labels[idx], scores[idx], boxes[idx]) for idx in indices.flatten()]
def draw_boxes(img: np.ndarray, boxes):
"""
Draw detected boxes on the image.
"""
for label, score, box in boxes:
# Choose color for the label.
color = tuple(map(int, colors[label]))
# Draw a box.
x2 = box[0] + box[2]
y2 = box[1] + box[3]
cv2.rectangle(img=img, pt1=box[:2], pt2=(x2, y2), color=color, thickness=2)
# Draw a label name inside the box.
cv2.putText(
img=img,
text=f"{classes[label]} {score:.2f}",
org=(box[0] + 10, box[1] + 20),
fontFace=cv2.FONT_HERSHEY_COMPLEX,
fontScale=img.shape[1] / 1200,
color=color,
thickness=1,
lineType=cv2.LINE_AA,
)
def show_result(results: np.ndarray):
"""
Postprocess the raw results, draw boxes and show the image.
"""
output_img = image.copy()
detections = postprocess(results)
draw_boxes(output_img, detections)
utils.show_array(output_img)
Optimizations `⇑ <#top>`__
###############################################################################################################################
Below, we present the performance tricks for faster inference in the
throughput mode. We release resources after every benchmarking to be
sure the same amount of resource is available for every experiment.
PyTorch model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
First, we’re benchmarking the original PyTorch model without any
optimizations applied. We will treat it as our baseline.
.. code:: ipython3
import torch
with torch.no_grad():
result = pytorch_model(torch.as_tensor(video_frames[0])).detach().numpy()[0]
show_result(result)
pytorch_fps = benchmark_model(pytorch_model, frames=torch.as_tensor(video_frames).float(), benchmark_name="PyTorch model")
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_14_0.jpg
.. parsed-literal::
PyTorch model on CPU. First inference time: 0.0266 seconds
PyTorch model on CPU: 0.0200 seconds per image (49.99 FPS)
OpenVINO IR model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The first optimization is exporting the PyTorch model to OpenVINO
Intermediate Representation (IR) FP16 and running it. Reducing the
precision is one of the well-known methods for faster inference provided
the hardware that supports lower precision, such as FP16 or even INT8.
If the hardware doesn’t support lower precision, the model will be
inferred in FP32 automatically. We could also use quantization (INT8),
but we should experience a little accuracy drop. That’s why we skip that
step in this notebook.
.. code:: ipython3
from openvino.tools import mo
onnx_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}").with_suffix(".onnx")
# export PyTorch model to ONNX if it doesn't already exist
if not onnx_path.exists():
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_path)
# convert ONNX model to IR, use FP16
ov_model = mo.convert_model(onnx_path, compress_to_fp16=True)
.. code:: ipython3
ov_cpu_model = core.compile_model(ov_model, device_name="CPU")
result = ov_cpu_model(video_frames[0])[ov_cpu_model.output(0)][0]
show_result(result)
ov_cpu_fps = benchmark_model(model=ov_cpu_model, frames=video_frames, benchmark_name="OpenVINO model")
del ov_cpu_model # release resources
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_17_0.jpg
.. parsed-literal::
OpenVINO model on CPU. First inference time: 0.0195 seconds
OpenVINO model on CPU: 0.0073 seconds per image (136.92 FPS)
OpenVINO IR model + bigger batch `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Batch processing often gives higher throughput as more inputs are
processed at once. To use bigger batches (than 1), we must convert the
model again, specifying a new input shape, and reshape input frames. In
our case, a batch size equal to 4 is the best choice, but optimal batch
size is very device-specific and depends on many factors, e.g.,
inference precision. We recommend trying various sizes for other
hardware and model.
.. code:: ipython3
batch_size = 4
onnx_batch_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}_batch_{batch_size}").with_suffix(".onnx")
if not onnx_batch_path.exists():
dummy_input = torch.randn(batch_size, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_batch_path)
# export the model with the bigger batch size
ov_batch_model = mo.convert_model(onnx_batch_path, compress_to_fp16=True)
.. parsed-literal::
/opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:514: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
/opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py:64: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:
.. code:: ipython3
ov_cpu_batch_model = core.compile_model(ov_batch_model, device_name="CPU")
batched_video_frames = video_frames.reshape([-1, batch_size, 3, IMAGE_HEIGHT, IMAGE_WIDTH])
result = ov_cpu_batch_model(batched_video_frames[0])[ov_cpu_batch_model.output(0)][0]
show_result(result)
ov_cpu_batch_fps = benchmark_model(model=ov_cpu_batch_model, frames=batched_video_frames, benchmark_name="OpenVINO model + bigger batch")
del ov_cpu_batch_model # release resources
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_20_0.jpg
.. parsed-literal::
OpenVINO model + bigger batch on CPU. First inference time: 0.0590 seconds
OpenVINO model + bigger batch on CPU: 0.0069 seconds per image (143.96 FPS)
OpenVINO IR model in throughput mode `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO allows specifying a performance hint changing the internal
configuration of the device. There are three different hints:
``LATENCY``, ``THROUGHPUT``, and ``CUMULATIVE_THROUGHPUT``. As this
notebook is focused on the throughput mode, we will use the latter two.
The hints can be used with other devices as well. Throughput mode
implicitly triggers using the `Automatic
Batching <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_Automatic_Batching.html>`__
feature, which sets the batch size to the optimal level.
.. code:: ipython3
ov_cpu_through_model = core.compile_model(ov_model, device_name="CPU", config={"PERFORMANCE_HINT": "THROUGHPUT"})
result = ov_cpu_through_model(video_frames[0])[ov_cpu_through_model.output(0)][0]
show_result(result)
ov_cpu_through_fps = benchmark_model(model=ov_cpu_through_model, frames=video_frames, benchmark_name="OpenVINO model", device_name="CPU (THROUGHPUT)")
del ov_cpu_through_model # release resources
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_22_0.jpg
.. parsed-literal::
OpenVINO model on CPU (THROUGHPUT). First inference time: 0.0226 seconds
OpenVINO model on CPU (THROUGHPUT): 0.0117 seconds per image (85.50 FPS)
OpenVINO IR model in throughput mode on GPU `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Usually, a GPU device provides more frames per second than a CPU, so
let’s run the above model on the GPU. Please note you need to have an
Intel GPU and `install
drivers <https://github.com/openvinotoolkit/openvino_notebooks/wiki/Ubuntu#1-install-python-git-and-gpu-drivers-optional>`__
to be able to run this step. In addition, offloading to the GPU helps
reduce CPU load and memory consumption, allowing it to be left for
routine processes. If you cannot observe a higher throughput on GPU, it
may be because the model is too light to benefit from massive parallel
execution.
.. code:: ipython3
ov_gpu_fps = 0.0
if "GPU" in core.available_devices:
# compile for GPU
ov_gpu_model = core.compile_model(ov_model, device_name="GPU", config={"PERFORMANCE_HINT": "THROUGHPUT"})
result = ov_gpu_model(video_frames[0])[ov_gpu_model.output(0)][0]
show_result(result)
ov_gpu_fps = benchmark_model(model=ov_gpu_model, frames=video_frames, benchmark_name="OpenVINO model", device_name="GPU (THROUGHPUT)")
del ov_gpu_model # release resources
OpenVINO IR model in throughput mode on AUTO `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OpenVINO offers a virtual device called
`AUTO <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__,
which can select the best device for us based on the aforementioned
performance hint.
.. code:: ipython3
ov_auto_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "THROUGHPUT"})
result = ov_auto_model(video_frames[0])[ov_auto_model.output(0)][0]
show_result(result)
ov_auto_fps = benchmark_model(model=ov_auto_model, frames=video_frames, benchmark_name="OpenVINO model", device_name="AUTO (THROUGHPUT)")
del ov_auto_model # release resources
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_26_0.jpg
.. parsed-literal::
OpenVINO model on AUTO (THROUGHPUT). First inference time: 0.0257 seconds
OpenVINO model on AUTO (THROUGHPUT): 0.0215 seconds per image (46.61 FPS)
OpenVINO IR model in cumulative throughput mode on AUTO `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The AUTO device in throughput mode will select the best, but one
physical device to bring the highest throughput. However, if we have
more Intel devices like CPU, iGPUs, and dGPUs in one machine, we may
benefit from them all. To do so, we must use cumulative throughput to
activate all devices.
.. code:: ipython3
ov_auto_cumulative_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "CUMULATIVE_THROUGHPUT"})
result = ov_auto_cumulative_model(video_frames[0])[ov_auto_cumulative_model.output(0)][0]
show_result(result)
ov_auto_cumulative_fps = benchmark_model(model=ov_auto_cumulative_model, frames=video_frames, benchmark_name="OpenVINO model", device_name="AUTO (CUMULATIVE THROUGHPUT)")
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_28_0.jpg
.. parsed-literal::
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT). First inference time: 0.0268 seconds
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT): 0.0216 seconds per image (46.25 FPS)
OpenVINO IR model in cumulative throughput mode on AUTO + asynchronous processing `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Asynchronous mode means that OpenVINO immediately returns from an
inference call and doesn’t wait for the result. It requires more
concurrent code to be written, but should offer better processing time
utilization e.g. we can run some pre- or post-processing code while
waiting for the result. Although we could use async processing directly
(start_async() function), it’s recommended to use AsyncInferQueue, which
is an easier approach to achieve the same outcome. This class
automatically spawns the pool of InferRequest objects (also called
“jobs”) and provides synchronization mechanisms to control the flow of
the pipeline.
**NOTE**: Asynchronous processing cannot guarantee outputs to be in
the same order as inputs, so be careful in case of applications when
the order of frames matters, e.g., videos.
.. code:: ipython3
from openvino.runtime import AsyncInferQueue
def callback(infer_request, info):
result = infer_request.get_output_tensor(0).data[0]
show_result(result)
pass
infer_queue = AsyncInferQueue(ov_auto_cumulative_model)
infer_queue.set_callback(callback) # set callback to post-process (show) results
infer_queue.start_async(video_frames[0])
infer_queue.wait_all()
# don't show output for the remaining frames
infer_queue.set_callback(lambda x, y: {})
ov_async_model = benchmark_model(model=infer_queue.start_async, frames=video_frames, async_queue=infer_queue, benchmark_name="OpenVINO model in asynchronous processing", device_name="AUTO (CUMULATIVE THROUGHPUT)")
del infer_queue # release resources
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_30_0.jpg
.. parsed-literal::
OpenVINO model in asynchronous processing on AUTO (CUMULATIVE THROUGHPUT). First inference time: 0.0239 seconds
OpenVINO model in asynchronous processing on AUTO (CUMULATIVE THROUGHPUT): 0.0041 seconds per image (245.46 FPS)
Other tricks `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
There are other tricks for performance improvement, such as advanced
options, quantization and pre-post-processing or dedicated to latency
mode. To get even more from your model, please visit `advanced
throughput
options <https://docs.openvino.ai/2023.0/openvino_docs_deployment_optimization_guide_tput_advanced.html>`__,
`109-latency-tricks <109-latency-tricks.ipynb>`__,
`111-detection-quantization <../111-detection-quantization>`__, and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
Performance comparison `⇑ <#top>`__
###############################################################################################################################
The following graphical comparison is valid for the selected model and
hardware simultaneously. If you cannot see any improvement between some
steps, just skip them.
.. code:: ipython3
%matplotlib inline
.. code:: ipython3
from matplotlib import pyplot as plt
labels = ["PyTorch model", "OpenVINO IR model", "OpenVINO IR model + bigger batch", "OpenVINO IR model in throughput mode", "OpenVINO IR model in throughput mode on GPU",
"OpenVINO IR model in throughput mode on AUTO", "OpenVINO IR model in cumulative throughput mode on AUTO", "OpenVINO IR model in cumulative throughput mode on AUTO + asynchronous processing"]
fps = [pytorch_fps, ov_cpu_fps, ov_cpu_batch_fps, ov_cpu_through_fps, ov_gpu_fps, ov_auto_fps, ov_auto_cumulative_fps, ov_async_model]
bar_colors = colors[::10] / 255.0
fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(labels, fps, color=bar_colors)
ax.set_ylabel("Throughput [FPS]")
ax.set_title("Performance difference")
plt.xticks(rotation='vertical')
plt.show()
.. image:: 109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_33_0.png
Conclusions `⇑ <#top>`__
###############################################################################################################################
We already showed the steps needed to improve the throughput of an
object detection model. Even if you experience much better performance
after running this notebook, please note this may not be valid for every
hardware or every model. For the most accurate results, please use
``benchmark_app`` `command-line
tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_samples_benchmark_app_README.html>`__.
Note that ``benchmark_app`` cannot measure the impact of some tricks
above.

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_14_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_17_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_20_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_22_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_26_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_28_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_30_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcd7da923bb0f72430eaf7b4770175320f1f3219aaca2d460c54fa9ef07e51c2
size 162756

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_33_0.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9547ead08b2efa00ac1380ab53f7cdb22f416ea13ecf0d9eb556703e9fcdde0d
size 77855

3
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_4_0.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:41c502fdff24ada81c63ccfca7d9153ea368b1eb3330caa03afa3281c35e4484
size 155828

15
docs/notebooks/109-throughput-tricks-with-output_files/index.html Normal file
View File

@ -0,0 +1,15 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/109-throughput-tricks-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/109-throughput-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-throughput-tricks-with-output_14_0.jpg">109-throughput-tricks-with-output_14_0.jpg</a> 16-Aug-2023 01:31 162715
<a href="109-throughput-tricks-with-output_17_0.jpg">109-throughput-tricks-with-output_17_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_20_0.jpg">109-throughput-tricks-with-output_20_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_22_0.jpg">109-throughput-tricks-with-output_22_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_26_0.jpg">109-throughput-tricks-with-output_26_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_28_0.jpg">109-throughput-tricks-with-output_28_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_30_0.jpg">109-throughput-tricks-with-output_30_0.jpg</a> 16-Aug-2023 01:31 162756
<a href="109-throughput-tricks-with-output_33_0.png">109-throughput-tricks-with-output_33_0.png</a> 16-Aug-2023 01:31 77855
<a href="109-throughput-tricks-with-output_4_0.jpg">109-throughput-tricks-with-output_4_0.jpg</a> 16-Aug-2023 01:31 155828
</pre><hr></body>
</html>

126
docs/notebooks/110-ct-scan-live-inference-with-output.rst
View File

@ -1,6 +1,8 @@
Live Inference and Benchmark CT-scan Data with OpenVINO™
========================================================
.. _top:
Kidney Segmentation with PyTorch Lightning and OpenVINO™ - Part 4
-----------------------------------------------------------------
@ -16,25 +18,39 @@ live inference with async API and MULTI plugin in OpenVINO.
This notebook needs a quantized OpenVINO IR model and images from the
`KiTS-19 <https://github.com/neheller/kits19>`__ dataset, converted to
2D images. (To learn how the model is quantized, see the `Convert and
Quantize a UNet Model and Show Live
Inference <110-ct-segmentation-quantize-nncf.ipynb>`__ tutorial.)
2D images. (To learn how the model is quantized, see the
`Convert and Quantize a UNet Model and Show Live Inference <110-ct-segmentation-quantize-nncf-with-output.html>`__ tutorial.)
This notebook provides a pre-trained model, trained for 20 epochs with
the full KiTS-19 frames dataset, which has an F1 score on the validation
set of 0.9. The training code is available in the `PyTorch Monai
Training <110-ct-segmentation-quantize-with-output.html>`__
set of 0.9. The training code is available in the
`PyTorch MONAI Training <110-ct-segmentation-quantize-with-output.html>`__
notebook.
For demonstration purposes, this tutorial will download one converted CT
scan to use for inference.
scan to use for inference.
**Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Benchmark Model Performance <#benchmark-model-performance>`__
- `Download and Prepare Data <#download-and-prepare-data>`__
- `Show Live Inference <#show-live-inference>`__
- `Load Model and List of Image Files <#load-model-and-list-of-image-files>`__
- `Prepare images <#prepare-images>`__
- `Specify device <#specify-device>`__
- `Setting callback function <#setting-callback-function>`__
- `Create asynchronous inference queue and perform it <#create-asynchronous-inference-queue-and-perform-it>`__
.. code:: ipython3
!pip install -q "monai>=0.9.1,<1.0.0"
Imports
-------
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -52,8 +68,9 @@ Imports
sys.path.append("../utils")
from notebook_utils import download_file
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
To use the pre-trained models, set ``IR_PATH`` to
``"pretrained_model/unet44.xml"`` and ``COMPRESSED_MODEL_PATH`` to
@ -90,11 +107,11 @@ trained or optimized yourself, adjust the model paths.
pretrained_model/quantized_unet_kits19.bin: 0%| | 0.00/1.90M [00:00<?, ?B/s]
Benchmark Model Performance
---------------------------
Benchmark Model Performance `⇑ <#top>`__
###############################################################################################################################
To measure the inference performance of the IR model, use `Benchmark
Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
To measure the inference performance of the IR model, use
`Benchmark Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
- an inference performance measurement tool in OpenVINO. Benchmark tool
is a command-line application that can be run in the notebook with
``! benchmark_app`` or ``%sx benchmark_app`` commands.
@ -157,7 +174,7 @@ is a command-line application that can be run in the notebook with
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 13.92 ms
[ INFO ] Read model took 13.69 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input.1 (node: input.1) : f32 / [...] / [1,1,512,512]
@ -171,7 +188,7 @@ is a command-line application that can be run in the notebook with
[ INFO ] Model outputs:
[ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 181.67 ms
[ INFO ] Compile model took 181.66 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
@ -200,21 +217,22 @@ is a command-line application that can be run in the notebook with
[ INFO ] Fill input 'input.1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 27.22 ms
[ INFO ] First inference took 26.05 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1431 iterations
[ INFO ] Duration: 15005.09 ms
[ INFO ] Count: 1424 iterations
[ INFO ] Duration: 15004.96 ms
[ INFO ] Latency:
[ INFO ] Median: 10.25 ms
[ INFO ] Average: 10.29 ms
[ INFO ] Min: 9.99 ms
[ INFO ] Max: 15.67 ms
[ INFO ] Throughput: 97.57 FPS
[ INFO ] Median: 10.29 ms
[ INFO ] Average: 10.35 ms
[ INFO ] Min: 10.14 ms
[ INFO ] Max: 14.72 ms
[ INFO ] Throughput: 97.14 FPS
Download and Prepare Data
-------------------------
Download and Prepare Data `⇑ <#top>`__
###############################################################################################################################
Download one validation video for live inference.
@ -260,8 +278,9 @@ downloaded and extracted in the next cell.
Downloaded and extracted data for case_00117
Show Live Inference
-------------------
Show Live Inference `⇑ <#top>`__
###############################################################################################################################
To show live inference on the model in the notebook, use the
asynchronous processing feature of OpenVINO Runtime.
@ -271,28 +290,31 @@ If you use a GPU device, with ``device="GPU"`` or
card, model loading will be slow the first time you run this code. The
model will be cached, so after the first time model loading will be
faster. For more information on OpenVINO Runtime, including Model
Caching, refer to the `OpenVINO API
tutorial <002-openvino-api-with-output.html>`__.
Caching, refer to the `OpenVINO API tutorial <002-openvino-api-with-output.html>`__.
We will use
`AsyncInferQueue <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_Python_API_exclusives.html#asyncinferqueue>`__
We will use `AsyncInferQueue <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_Python_API_exclusives.html#asyncinferqueue>`__
to perform asynchronous inference. It can be instantiated with compiled
model and a number of jobs - parallel execution threads. If you don’t
pass a number of jobs or pass ``0``, then OpenVINO will pick the optimal
number based on your device and heuristics. After acquiring the
inference queue, there are two jobs to do:
- Preprocess the data and push it to the inference queue. The preprocessing steps will remain the same.
- Tell the inference queue what to do with the model output after the inference is finished. It is represented by the ``callback`` python function that takes an inference result and data that we passed to the inference queue along with the prepared input data
- Preprocess the data and push it to the inference queue. The
preprocessing steps will remain the same.
- Tell the inference queue what to do with the model output after the
inference is finished. It is represented by the ``callback`` python
function that takes an inference result and data that we passed to
the inference queue along with the prepared input data
Everything else will be handled by the ``AsyncInferQueue`` instance.
Load Model and List of Image Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load Model and List of Image Files `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load the segmentation model to OpenVINO Runtime with
``SegmentationModel``, based on the Model API from `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__. This model
``SegmentationModel``, based on the Model API from
`Open Model Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__. This model
implementation includes pre and post processing for the model. For
``SegmentationModel`` this includes the code to create an overlay of the
segmentation mask on the original image/frame. Uncomment the next cell
@ -314,8 +336,9 @@ to see the implementation.
case_00117, 69 images
Preapre images
~~~~~~~~~~~~~~
Prepare images `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use the ``reader = LoadImage()`` function to read the images in the same
way as in the
@ -335,8 +358,9 @@ tutorial.
framebuf.append(image)
next_frame_id += 1
Specify device
~~~~~~~~~~~~~~
Specify device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -351,14 +375,15 @@ Specify device
Setting callback function
~~~~~~~~~~~~~~~~~~~~~~~~~
Setting callback function `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When ``callback`` is set, any job that ends the inference, calls the
Python function. The ``callback`` function must have two arguments: one
is the request that calls the ``callback``, which provides the
InferRequest API; the other is called “userdata”, which provides the
possibility of passing runtime values.
``InferRequest`` API; the other is called ``userdata``, which provides
the possibility of passing runtime values.
The ``callback`` function will show the results of inference.
@ -387,8 +412,9 @@ The ``callback`` function will show the results of inference.
display.clear_output(wait=True)
display.display(i)
Create asynchronous inference queue and perform it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create asynchronous inference queue and perform it `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -433,6 +459,6 @@ Create asynchronous inference queue and perform it
.. parsed-literal::
Loaded model to Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') in 0.18 seconds.
Total time to infer all frames: 3.416s
Time per frame: 0.050229s (19.909 FPS)
Total time to infer all frames: 3.401s
Time per frame: 0.050022s (19.991 FPS)

6
docs/notebooks/110-ct-scan-live-inference-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/110-ct-scan-live-inference-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/110-ct-scan-live-inference-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/110-ct-scan-live-inference-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-scan-live-inference-with-output_21_0.png">110-ct-scan-live-inference-with-output_21_0.png</a> 12-Jul-2023 00:11 48780
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/110-ct-scan-live-inference-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-scan-live-inference-with-output_21_0.png">110-ct-scan-live-inference-with-output_21_0.png</a> 16-Aug-2023 01:31 48780
</pre><hr></body>
</html>

260
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output.rst
View File

@ -1,6 +1,8 @@
Quantize a Segmentation Model and Show Live Inference
=====================================================
.. _top:
Kidney Segmentation with PyTorch Lightning and OpenVINO™ - Part 3
-----------------------------------------------------------------
@ -13,10 +15,8 @@ scratch; the data is from
This third tutorial in the series shows how to:
- Convert an Original model to OpenVINO IR with `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__,
using `Model Optimizer Python
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
- Convert an Original model to OpenVINO IR with `model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
- Quantize a PyTorch model with NNCF
- Evaluate the F1 score metric of the original model and the quantized
model
@ -48,19 +48,46 @@ NNCF for PyTorch models requires a C++ compiler. On Windows, install
2019 <https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes>`__.
During installation, choose Desktop development with C++ in the
Workloads tab. On macOS, run ``xcode-select –install`` from a Terminal.
On Linux, install gcc.
On Linux, install ``gcc``.
Running this notebook with the full dataset will take a long time. For
demonstration purposes, this tutorial will download one converted CT
scan and use that scan for quantization and inference. For production
purposes, use a representative dataset for quantizing the model.
purposes, use a representative dataset for quantizing the model.
**Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Load PyTorch Model <#load-pytorch-model>`__
- `Download CT-scan Data <#download-ct-scan-data>`__
- `Configuration <#configuration>`__
- `Dataset <#dataset>`__
- `Metric <#metric>`__
- `Quantization <#quantization>`__
- `Compare FP32 and INT8 Model <#compare-fp32-and-int8-model>`__
- `Compare File Size <#compare-file-size>`__
- `Compare Metrics for the original model and the quantized model to be sure that there no degradation. <#compare-metrics-for-the-original-model-and-the-quantized-model-to-be-sure-that-there-no-degradation>`__
- `Compare Performance of the FP32 IR Model and Quantized Models <#compare-performance-of-the-fp32-ir-model-and-quantized-models>`__
- `Visually Compare Inference Results <#visually-compare-inference-results>`__
- `Show Live Inference <#show-live-inference>`__
- `Load Model and List of Image Files <#load-model-and-list-of-image-files>`__
- `Show Inference <#show-inference>`__
- `References <#references>`__
.. code:: ipython3
!pip install -q "monai>=0.9.1,<1.0.0" "torchmetrics>=0.11.0"
Imports
-------
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -146,10 +173,10 @@ Imports
.. parsed-literal::
2023-07-11 22:37:38.702892: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:37:38.737776: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:41:33.627938: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:41:33.662730: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:37:39.282688: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:41:34.189615: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -157,8 +184,9 @@ Imports
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
By default, this notebook will download one CT scan from the KITS19
dataset that will be used for quantization. To use the full dataset, set
@ -173,16 +201,16 @@ dataset that will be used for quantization. To use the full dataset, set
MODEL_DIR = Path("model")
MODEL_DIR.mkdir(exist_ok=True)
Load PyTorch Model
------------------
Load PyTorch Model `⇑ <#top>`__
###############################################################################################################################
Download the pre-trained model weights, load the PyTorch model and the
``state_dict`` that was saved after training. The model used in this
notebook is a
`BasicUnet <https://docs.monai.io/en/stable/networks.html#basicunet>`__
notebook is a `BasicUNet <https://docs.monai.io/en/stable/networks.html#basicunet>`__
model from `MONAI <https://monai.io>`__. We provide a pre-trained
checkpoint. To see how this model performs, check out the `training
notebook <pytorch-monai-training.ipynb>`__.
checkpoint. To see how this model performs, check out the
`training notebook <pytorch-monai-training.ipynb>`__.
.. code:: ipython3
@ -219,8 +247,9 @@ notebook <pytorch-monai-training.ipynb>`__.
Download CT-scan Data
---------------------
Download CT-scan Data `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -245,19 +274,20 @@ Download CT-scan Data
Data for case_00117 exists
Configuration
-------------
Configuration `⇑ <#top>`__
###############################################################################################################################
Dataset
~~~~~~~
The KitsDataset class in the next cell expects images and masks in the
*basedir* directory, in a folder per patient. It is a simplified version
of the DataSet class in the `training
notebook <pytorch-monai-training.ipynb>`__.
Dataset `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The ``KitsDataset`` class in the next cell expects images and masks in
the *``basedir``* directory, in a folder per patient. It is a simplified
version of the Dataset class in the `training notebook <pytorch-monai-training.ipynb>`__.
Images are loaded with MONAI’s
```LoadImage`` <https://docs.monai.io/en/stable/transforms.html#loadimage>`__,
`LoadImage <https://docs.monai.io/en/stable/transforms.html#loadimage>`__,
to align with the image loading method in the training notebook. This
method rotates and flips the images. We define a ``rotate_and_flip``
method to display the images in the expected orientation:
@ -346,8 +376,8 @@ kidney pixels to verify that the annotations look correct:
.. image:: 110-ct-segmentation-quantize-nncf-with-output_files/110-ct-segmentation-quantize-nncf-with-output_15_1.png
Metric
~~~~~~
Metric `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Define a metric to determine the performance of the model.
@ -385,8 +415,9 @@ library.
metric.update(label.flatten(), prediction.flatten())
return metric.compute()
Quantization
------------
Quantization `⇑ <#top>`__
###############################################################################################################################
Before quantizing the model, we compute the F1 score on the ``FP32``
model, for comparison:
@ -416,26 +447,27 @@ this notebook.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
`NNCF <https://github.com/openvinotoolkit/nncf>`__ provides a suite of
advanced algorithms for Neural Networks inference optimization in
OpenVINO with minimal accuracy drop.
OpenVINO with minimal accuracy drop.
.. note::
NNCF Post-training Quantization is available in OpenVINO 2023.0 release.
**Note**: NNCF Post-training Quantization is available in OpenVINO
2023.0 release.
Create a quantized model from the pre-trained ``FP32`` model and the
calibration dataset. The optimization process contains the following
steps:
1. Create a Dataset for quantization.
2. Run ``nncf.quantize`` for getting an optimized model.
3. Export the quantized model to ONNX and then convert to OpenVINO IR model.
4. Serialize the INT8 model using ``openvino.runtime.serialize`` function for benchmarking.
::
1. Create a Dataset for quantization.
2. Run `nncf.quantize` for getting an optimized model.
3. Export the quantized model to ONNX and then convert to OpenVINO IR model.
4. Serialize the INT8 model using `openvino.runtime.serialize` function for benchmarking.
.. code:: ipython3
@ -479,28 +511,29 @@ model and save it.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:338: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:338: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:346: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:346: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/quantize_functions.py:140: FutureWarning: 'torch.onnx._patch_torch._graph_op' is deprecated in version 1.13 and will be removed in version 1.14. Please note 'g.op()' is to be removed from torch.Graph. Please open a GitHub issue if you need this functionality..
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/quantize_functions.py:140: FutureWarning: 'torch.onnx._patch_torch._graph_op' is deprecated in version 1.13 and will be removed in version 1.14. Please note 'g.op()' is to be removed from torch.Graph. Please open a GitHub issue if you need this functionality..
output = g.op(
This notebook demonstrates post-training quantization with NNCF.
NNCF also supports quantization-aware training, and other algorithms
than quantization. See the `NNCF
documentation <https://github.com/openvinotoolkit/nncf/>`__ in the NNCF
than quantization. See the `NNCF documentation <https://github.com/openvinotoolkit/nncf/>`__ in the NNCF
repository for more information.
Compare FP32 and INT8 Model
---------------------------
Compare FP32 and INT8 Model `⇑ <#top>`__
###############################################################################################################################
Compare File Size `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Compare File Size
~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -517,8 +550,8 @@ Compare File Size
INT8 model size: 1953.49 KB
Compare Metrics for the original model and the quantized model to be sure that there no degradation.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compare Metrics for the original model and the quantized model to be sure that there no degradation. `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -537,12 +570,11 @@ Compare Metrics for the original model and the quantized model to be sure that t
INT8 F1: 0.999
Compare Performance of the FP32 IR Model and Quantized Models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compare Performance of the FP32 IR Model and Quantized Models `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To measure the inference performance of the ``FP32`` and ``INT8``
models, we use `Benchmark
Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
models, we use `Benchmark Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
- OpenVINO’s inference performance measurement tool. Benchmark tool is a
command line application, part of OpenVINO development tools, that can
be run in the notebook with ``! benchmark_app`` or
@ -586,7 +618,7 @@ be run in the notebook with ``! benchmark_app`` or
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 22.47 ms
[ INFO ] Read model took 21.45 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] 1 , x (node: Parameter_2) : f32 / [...] / [1,1,512,512]
@ -600,7 +632,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] 169 (node: aten::_convolution_861) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 89.66 ms
[ INFO ] Compile model took 87.17 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -622,17 +654,17 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Fill input '1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 56.61 ms
[ INFO ] First inference took 55.25 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 427 iterations
[ INFO ] Duration: 15001.35 ms
[ INFO ] Count: 425 iterations
[ INFO ] Duration: 15001.92 ms
[ INFO ] Latency:
[ INFO ] Median: 34.89 ms
[ INFO ] Average: 34.92 ms
[ INFO ] Min: 34.57 ms
[ INFO ] Max: 39.09 ms
[ INFO ] Throughput: 28.66 FPS
[ INFO ] Median: 35.01 ms
[ INFO ] Average: 35.08 ms
[ INFO ] Min: 34.54 ms
[ INFO ] Max: 37.24 ms
[ INFO ] Throughput: 28.56 FPS
.. code:: ipython3
@ -658,7 +690,7 @@ be run in the notebook with ``! benchmark_app`` or
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 32.41 ms
[ INFO ] Read model took 33.80 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x.1 (node: x.1) : f32 / [...] / [1,1,512,512]
@ -672,7 +704,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] 578 (node: 578) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 144.74 ms
[ INFO ] Compile model took 144.48 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -694,21 +726,22 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Fill input 'x.1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 32.29 ms
[ INFO ] First inference took 30.85 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 985 iterations
[ INFO ] Duration: 15013.03 ms
[ INFO ] Count: 973 iterations
[ INFO ] Duration: 15001.74 ms
[ INFO ] Latency:
[ INFO ] Median: 14.99 ms
[ INFO ] Average: 15.04 ms
[ INFO ] Min: 14.76 ms
[ INFO ] Max: 17.68 ms
[ INFO ] Throughput: 66.70 FPS
[ INFO ] Median: 15.17 ms
[ INFO ] Average: 15.21 ms
[ INFO ] Min: 14.84 ms
[ INFO ] Max: 17.66 ms
[ INFO ] Throughput: 65.90 FPS
Visually Compare Inference Results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Visually Compare Inference Results `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Visualize the results of the model on four slices of the validation set.
Compare the results of the ``FP32`` IR model with the results of the
@ -787,24 +820,23 @@ seed is displayed to enable reproducing specific runs of this cell.
.. parsed-literal::
Visualizing results with seed 1689107962
Visualizing results with seed 1692132195
.. image:: 110-ct-segmentation-quantize-nncf-with-output_files/110-ct-segmentation-quantize-nncf-with-output_37_1.png
Show Live Inference
-------------------
Show Live Inference `⇑ <#top>`__
###############################################################################################################################
To show live inference on the model in the notebook, we will use the
asynchronous processing feature of OpenVINO.
We use the ``show_live_inference`` function from `Notebook
Utils <utils-with-output.html>`__ to show live inference. This
function uses `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__\ ’s
AsyncPipeline and Model API to perform asynchronous inference. After
We use the ``show_live_inference`` function from `Notebook Utils <utils-with-output.html>`__ to show live inference. This
function uses `Open Model Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__
Async Pipeline and Model API to perform asynchronous inference. After
inference on the specified CT scan has completed, the total time and
throughput (fps), including preprocessing and displaying, will be
printed.
@ -812,12 +844,12 @@ printed.
**NOTE**: If you experience flickering on Firefox, consider using
Chrome or Edge to run this notebook.
Load Model and List of Image Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load Model and List of Image Files `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
We load the segmentation model to OpenVINO Runtime with
``SegmentationModel``, based on the `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ Model API.
``SegmentationModel``, based on the `Open Model Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ Model API.
This model implementation includes pre and post processing for the
model. For ``SegmentationModel``, this includes the code to create an
overlay of the segmentation mask on the original image/frame.
@ -839,8 +871,9 @@ overlay of the segmentation mask on the original image/frame.
case_00117, 69 images
Show Inference
~~~~~~~~~~~~~~
Show Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In the next cell, we run the ``show_live_inference`` function, which
loads the ``segmentation_model`` to the specified ``device`` (using
@ -864,27 +897,24 @@ performs inference, and displays the results on the frames loaded in
.. parsed-literal::
Loaded model to CPU in 0.13 seconds.
Total time for 68 frames: 3.28 seconds, fps:21.05
Loaded model to CPU in 0.12 seconds.
Total time for 68 frames: 3.35 seconds, fps:20.60
References
----------
References `⇑ <#top>`__
###############################################################################################################################
**OpenVINO** - `NNCF
Repository <https://github.com/openvinotoolkit/nncf/>`__ - `Neural
Network Compression Framework for fast model
inference <https://arxiv.org/abs/2002.08679>`__ - `OpenVINO API
Tutorial <002-openvino-api-with-output.html>`__ - `OpenVINO
PyPI (pip install
openvino-dev) <https://pypi.org/project/openvino-dev/>`__
**Kits19 Data** - `Kits19 Challenge
Homepage <https://kits19.grand-challenge.org/>`__ - `Kits19 Github
Repository <https://github.com/neheller/kits19>`__ - `The KiTS19
Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT
Semantic Segmentations, and Surgical
Outcomes <https://arxiv.org/abs/1904.00445>`__ - `The state of the art
in kidney and kidney tumor segmentation in contrast-enhanced CT imaging:
Results of the KiTS19
challenge <https://www.sciencedirect.com/science/article/pii/S1361841520301857>`__
**OpenVINO**
- `NNCF Repository <https://github.com/openvinotoolkit/nncf/>`__
- `Neural Network Compression Framework for fast model inference <https://arxiv.org/abs/2002.08679>`__
- `OpenVINO API Tutorial <002-openvino-api-with-output.html>`__
- `OpenVINO PyPI (pip install openvino-dev) <https://pypi.org/project/openvino-dev/>`__
**Kits19 Data**
- `Kits19 Challenge Homepage <https://kits19.grand-challenge.org/>`__
- `Kits19 GitHub Repository <https://github.com/neheller/kits19>`__
- `The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes <https://arxiv.org/abs/1904.00445>`__
- `The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge <https://www.sciencedirect.com/science/article/pii/S1361841520301857>`__

4
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output_files/110-ct-segmentation-quantize-nncf-with-output_37_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:960b979d5c041593481e188c2f1f14c72c314c8b2338c263ad450e6ad52e3600
size 385435
oid sha256:aee7d177b413f4c198de3e7c40f1624d59abbdabe61e8ec20798b4649ac46f43
size 383352

10
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output_files/index.html
View File

@ -1,9 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-segmentation-quantize-nncf-with-output_15_1.png">110-ct-segmentation-quantize-nncf-with-output_1..&gt;</a> 12-Jul-2023 00:11 158997
<a href="110-ct-segmentation-quantize-nncf-with-output_37_1.png">110-ct-segmentation-quantize-nncf-with-output_3..&gt;</a> 12-Jul-2023 00:11 385435
<a href="110-ct-segmentation-quantize-nncf-with-output_42_0.jpg">110-ct-segmentation-quantize-nncf-with-output_4..&gt;</a> 12-Jul-2023 00:11 73812
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-segmentation-quantize-nncf-with-output_15_1.png">110-ct-segmentation-quantize-nncf-with-output_1..&gt;</a> 16-Aug-2023 01:31 158997
<a href="110-ct-segmentation-quantize-nncf-with-output_37_1.png">110-ct-segmentation-quantize-nncf-with-output_3..&gt;</a> 16-Aug-2023 01:31 383352
<a href="110-ct-segmentation-quantize-nncf-with-output_42_0.jpg">110-ct-segmentation-quantize-nncf-with-output_4..&gt;</a> 16-Aug-2023 01:31 73812
</pre><hr></body>
</html>

336
docs/notebooks/111-yolov5-quantization-migration-with-output.rst
View File

@ -1,16 +1,15 @@
Migrate quantization from POT API to NNCF API
=============================================
.. _top:
This tutorial demonstrates how to migrate quantization pipeline written
using the OpenVINO `Post-Training Optimization Tool
(POT) <https://docs.openvino.ai/2023.0/pot_introduction.html>`__ to
`NNCF Post-Training Quantization
API <https://docs.openvino.ai/nightly/basic_quantization_flow.html>`__.
This tutorial is based on `Ultralytics
Yolov5 <https://github.com/ultralytics/yolov5>`__ model and additionally
using the OpenVINO `Post-Training Optimization Tool (POT) <https://docs.openvino.ai/2023.0/pot_introduction.html>`__ to
`NNCF Post-Training Quantization API <https://docs.openvino.ai/2023.0/basic_quantization_flow.html>`__.
This tutorial is based on `Ultralytics YOLOv5 <https://github.com/ultralytics/yolov5>`__ model and additionally
it compares model accuracy between the FP32 precision and quantized INT8
precision models and runs a demo of model inference based on sample code
from `Ultralytics Yolov5 <https://github.com/ultralytics/yolov5>`__ with
from `Ultralytics YOLOv5 <https://github.com/ultralytics/yolov5>`__ with
the OpenVINO backend.
The tutorial consists from the following parts:
@ -21,17 +20,48 @@ The tutorial consists from the following parts:
4. Perform model optimization.
5. Compare accuracy FP32 and INT8 models
6. Run model inference demo
7. Compare performance FP32 and INt8 models
7. Compare performance FP32 and INT8 models
Preparation
-----------
Download the YOLOv5 model
~~~~~~~~~~~~~~~~~~~~~~~~~
**Table of contents**:
- `Preparation <#preparation>`__
- `Download the YOLOv5 model <#download-the-yolov5-model>`__
- `Conversion of the YOLOv5 model to OpenVINO <#conversion-of-the-yolov5-model-to-openvino>`__
- `Imports <#imports>`__
- `Prepare dataset for quantization <#prepare-dataset-for-quantization>`__
- `Create YOLOv5 DataLoader class for POT <#create-yolov5-dataloader-class-for-pot>`__
- `Create NNCF Dataset <#create-nncf-dataset>`__
- `Configure quantization pipeline <#configure-quantization-pipeline>`__
- `Prepare config and pipeline for POT <#prepare-config-and-pipeline-for-pot>`__
- `Prepare configuration parameters for NNCF <#prepare-configuration-parameters-for-nncf>`__
- `Perform model optimization <#perform-model-optimization>`__
- `Run quantization using POT <#run-quantization-using-pot>`__
- `Run quantization using NNCF <#run-quantization-using-nncf>`__
- `Compare accuracy FP32 and INT8 models <#compare-accuracy-fp32-and-int8-models>`__
- `Inference Demo Performance Comparison <#inference-demo-performance-comparison>`__
- `Benchmark <#benchmark>`__
- `References <#references>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Download the YOLOv5 model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
!pip install -q 'openvino-dev>=2023.0.0' 'nncf>=2.5.0'
!pip install -q "openvino-dev>=2023.0.0" "nncf>=2.5.0"
!pip install -q psutil "seaborn>=0.11.0" matplotlib numpy onnx
.. code:: ipython3
@ -63,24 +93,26 @@ Download the YOLOv5 model
``git clone https://github.com/ultralytics/yolov5.git -b v7.0``
Conversion of the YOLOv5 model to OpenVINO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Conversion of the YOLOv5 model to OpenVINO `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
There are three variables provided for easy run through all the notebook
cells.
There are three variables provided for easy run through all the notebook cells.
* ``IMAGE_SIZE`` - the image size for model input.
* ``MODEL_NAME`` - the model you want to use. It can be either yolov5s, yolov5m or yolov5l and so on.
* ``MODEL_PATH`` - to the path of the model directory in the YOLOv5 repository.
- ``IMAGE_SIZE`` - the image size for model input.
- ``MODEL_NAME`` - the model you want to use. It can be either yolov5s,
yolov5m or yolov5l and so on.
- ``MODEL_PATH`` - to the path of the model directory in the YOLOv5
repository.
YoloV5 ``export.py`` scripts support multiple model formats for
YOLOv5 ``export.py`` scripts support multiple model formats for
conversion. ONNX is also represented among supported formats. We need to
specify ``--include ONNX`` parameter for exporting. As the result,
directory with the ``{MODEL_NAME}`` name will be created with the
following content:
following content:
* ``{MODEL_NAME}.pt`` - the downloaded pre-trained weight.
* ``{MODEL_NAME}.onnx`` - the Open Neural Network Exchange (ONNX) is an open format, built to represent machine learning models.
- ``{MODEL_NAME}.pt`` - the downloaded pre-trained weight.
- ``{MODEL_NAME}.onnx`` - the Open Neural Network Exchange (ONNX) is an
open format, built to represent machine learning models.
.. code:: ipython3
@ -111,7 +143,7 @@ following content:
YOLOv5 🚀 v7.0-0-g915bbf2 Python-3.8.10 torch-1.13.1+cpu CPU
Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5m.pt to yolov5m/yolov5m.pt...
100%|██████████████████████████████████████| 40.8M/40.8M [00:09<00:00, 4.52MB/s]
100%|██████████████████████████████████████| 40.8M/40.8M [00:10<00:00, 4.02MB/s]
Fusing layers...
YOLOv5m summary: 290 layers, 21172173 parameters, 0 gradients
@ -119,10 +151,10 @@ following content:
PyTorch: starting from yolov5m/yolov5m.pt with output shape (1, 25200, 85) (40.8 MB)
ONNX: starting export with onnx 1.14.0...
ONNX: export success ✅ 1.2s, saved as yolov5m/yolov5m.onnx (81.2 MB)
ONNX: export success ✅ 1.3s, saved as yolov5m/yolov5m.onnx (81.2 MB)
Export complete (12.2s)
Results saved to /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/yolov5m
Export complete (13.7s)
Results saved to /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/yolov5m
Detect: python detect.py --weights yolov5m/yolov5m.onnx
Validate: python val.py --weights yolov5m/yolov5m.onnx
PyTorch Hub: model = torch.hub.load('ultralytics/yolov5', 'custom', 'yolov5m/yolov5m.onnx')
@ -130,17 +162,17 @@ following content:
Convert the ONNX model to OpenVINO Intermediate Representation (IR)
model generated by `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html#doxid-openvino-docs-m-o-d-g-deep-learning-model-optimizer-dev-guide>`__.
We will use `Model Optimizer Python
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
``openvino.tools.mo.convert_model`` function to convert ONNX model to
OpenVINO Model, then it can be seralized using
``openvino.runtime.serialize``\ As the result, directory with the
``{MODEL_DIR}`` name will be created with the following content:
* ``{MODEL_NAME}_fp32.xml``, ``{MODEL_NAME}_fp32.bin`` - OpenVINO Intermediate Representation (IR) model format with FP32 precision generated by Model Optimizer.
* ``{MODEL_NAME}_fp16.xml``, ``{MODEL_NAME}_fp16.bin`` - OpenVINO Intermediate Representation (IR) model format with FP32 precision generated by Model Optimizer.
model generated by `model conversion API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
We will use the ``openvino.tools.mo.convert_model`` function of model
conversion Python API to convert ONNX model to OpenVINO Model, then it
can be serialized using ``openvino.runtime.serialize``. As the result,
directory with the ``{MODEL_DIR}`` name will be created with the
following content: \* ``{MODEL_NAME}_fp32.xml``,
``{MODEL_NAME}_fp32.bin`` - OpenVINO Intermediate Representation (IR)
model format with FP32 precision generated by Model Optimizer. \*
``{MODEL_NAME}_fp16.xml``, ``{MODEL_NAME}_fp16.bin`` - OpenVINO
Intermediate Representation (IR) model format with FP32 precision
generated by Model Optimizer.
.. code:: ipython3
@ -170,8 +202,9 @@ OpenVINO Model, then it can be seralized using
Export ONNX to OpenVINO FP16 IR to: yolov5/yolov5m/FP16_openvino_model/yolov5m_fp16.xml
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -180,12 +213,13 @@ Imports
from yolov5.utils.dataloaders import create_dataloader
from yolov5.utils.general import check_dataset
Prepare dataset for quantization
--------------------------------
Prepare dataset for quantization `⇑ <#top>`__
###############################################################################################################################
Before starting quantization, we should prepare dataset, which will be
used for quantization. Ultralytics YOLOv5 provides data loader for
iteration overdataset during training and validation. Let’s create it
iteration over dataset during training and validation. Let’s create it
first.
.. code:: ipython3
@ -229,12 +263,13 @@ first.
.. parsed-literal::
Unzipping datasets/coco128.zip...
Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
New cache created: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache
Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
New cache created: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache
Create YOLOv5 DataLoader class for POT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create YOLOv5 DataLoader class for POT `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Create a class for loading the YOLOv5 dataset and annotation which
inherits from POT API class DataLoader.
@ -245,15 +280,15 @@ index. Any implementation should override the following methods:
- The ``__len__()``, returns the size of the dataset.
- The ``__getitem__()``, provides access to the data by index in range
of 0 to len(self). It can also encapsulate the logic of
of 0 to ``len(self)``. It can also encapsulate the logic of
model-specific pre-processing. This method should return data in the
(data, annotation) format, in which:
- The ``data`` is the input that is passed to the model at inference
so that it should be properly preprocessed. It can be either the
numpy.array object or a dictionary, where the key is the name of
the model input and value is numpy.array which corresponds to this
input.
``numpy.array`` object or a dictionary, where the key is the name
of the model input and value is ``numpy.array`` which corresponds
to this input.
- The ``annotation`` is not used by the Default Quantization method.
Therefore, this object can be None in this case.
@ -304,7 +339,7 @@ index. Any implementation should override the following methods:
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/offline_transformations/__init__.py:10: FutureWarning: The module is private and following namespace `offline_transformations` will be removed in the future.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/offline_transformations/__init__.py:10: FutureWarning: The module is private and following namespace `offline_transformations` will be removed in the future.
warnings.warn(
@ -322,15 +357,16 @@ index. Any implementation should override the following methods:
Nevergrad package could not be imported. If you are planning to use any hyperparameter optimization algo, consider installing it using pip. This implies advanced usage of the tool. Note that nevergrad is compatible only with Python 3.7+
Create NNCF Dataset
~~~~~~~~~~~~~~~~~~~
Create NNCF Dataset `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For preparing quantization dataset for NNCF, we should wrap
framework-specific data source into ``nncf.Dataset`` instance.
Additionaly, to transform data into model expected format we can define
Additionally, to transform data into model expected format we can define
transformation function, which accept data item for single dataset
iteration and transform it for feeding into model (e.g. in simpliest
case, if data item contains input tensor and anntation, we should
iteration and transform it for feeding into model (e.g. in simplest
case, if data item contains input tensor and annotation, we should
extract only input data from it and convert it into model expected
format).
@ -361,13 +397,15 @@ format).
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
Configure quantization pipeline
-------------------------------
Configure quantization pipeline `⇑ <#top>`__
###############################################################################################################################
Next, we should define quantization algorithm parameters.
Prepare config and pipeline for POT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prepare config and pipeline for POT `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
in POT, all quantization parameters should be defined using
configuration dictionary. Config consists of 3 sections: ``algorithms``
@ -415,12 +453,13 @@ pipeline using ``create_pipeline`` function.
# Step 5: Create a pipeline of compression algorithms.
pipeline = create_pipeline(algorithms_config, engine)
Prapare configuration parameters for NNCF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prepare configuration parameters for NNCF `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Post-training quantization pipeline in NNCF represented by
``nncf.quantize`` function for DefaultQuantization Algorithm and
``nncf.quantize_with_accuracy_control`` for AccuracyAwareQuantization.
``nncf.quantize`` function for Default Quantization Algorithm and
``nncf.quantize_with_accuracy_control`` for Accuracy Aware Quantization.
Quantization parameters ``preset``, ``model_type``, ``subset_size``,
``fast_bias_correction``, ``ignored_scope`` are arguments of function.
More details about supported parameters and formats can be found in NNCF
@ -435,11 +474,13 @@ in our case ``openvino.runtime.Model`` instance created using
subset_size = 300
preset = nncf.QuantizationPreset.MIXED
Perform model optimization
--------------------------
Perform model optimization `⇑ <#top>`__
###############################################################################################################################
Run quantization using POT `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Run quantization using POT
~~~~~~~~~~~~~~~~~~~~~~~~~~
To start model quantization using POT API, we should call
``pipeline.run(pot_model)`` method. As the result, we got quantized
@ -459,13 +500,14 @@ size of final .bin file.
save_model(compressed_model, optimized_save_dir, model_config["model_name"] + "_int8")
pot_int8_path = f"{optimized_save_dir}/{MODEL_NAME}_int8.xml"
Run quantization using NNCF
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run quantization using NNCF `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To run NNCF quantization, we should call ``nncf.quantize`` function. As
the result, the function returns quantized model in the same format like
input model, so it means that quantized model ready to be compiled on
device for inferece and can be saved on disk using
device for inference and can be saved on disk using
``openvino.runtime.serialize``.
.. code:: ipython3
@ -483,16 +525,17 @@ device for inferece and can be saved on disk using
.. parsed-literal::
Statistics collection: 43%|████▎ | 128/300 [00:30<00:41, 4.17it/s]
Biases correction: 100%|██████████| 82/82 [00:10<00:00, 7.86it/s]
Statistics collection: 43%|████▎ | 128/300 [00:30<00:40, 4.20it/s]
Biases correction: 100%|██████████| 82/82 [00:10<00:00, 7.71it/s]
Compare accuracy FP32 and INT8 models
-------------------------------------
Compare accuracy FP32 and INT8 models `⇑ <#top>`__
###############################################################################################################################
For getting accuracy results, we will use ``yolov5.val.run`` function
which already supports OpenVINO backend. For making int8 model is
compatible with Ultralytics provided validation pipeline, we alse should
compatible with Ultralytics provided validation pipeline, we also should
provide metadata with information about supported class names in the
same directory, where model located.
@ -551,10 +594,10 @@ same directory, where model located.
.. parsed-literal::
Forcing --batch-size 1 square inference (1,3,640,640) for non-PyTorch models
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
Class Images Instances P R mAP50 mAP50-95: 100%|██████████| 128/128 00:05
all 128 929 0.726 0.687 0.769 0.554
Speed: 0.2ms pre-process, 35.3ms inference, 3.2ms NMS per image at shape (1, 3, 640, 640)
Speed: 0.2ms pre-process, 35.3ms inference, 3.0ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5/runs/val/exp
@ -598,10 +641,10 @@ same directory, where model located.
.. parsed-literal::
Forcing --batch-size 1 square inference (1,3,640,640) for non-PyTorch models
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
Class Images Instances P R mAP50 mAP50-95: 100%|██████████| 128/128 00:03
all 128 929 0.761 0.677 0.773 0.548
Speed: 0.2ms pre-process, 17.3ms inference, 3.3ms NMS per image at shape (1, 3, 640, 640)
Speed: 0.2ms pre-process, 17.1ms inference, 3.3ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5/runs/val/exp2
@ -645,10 +688,10 @@ same directory, where model located.
.. parsed-literal::
Forcing --batch-size 1 square inference (1,3,640,640) for non-PyTorch models
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
val: Scanning /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/datasets/coco128/labels/train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 00:00
Class Images Instances P R mAP50 mAP50-95: 100%|██████████| 128/128 00:03
all 128 929 0.742 0.684 0.766 0.546
Speed: 0.2ms pre-process, 17.1ms inference, 3.3ms NMS per image at shape (1, 3, 640, 640)
Speed: 0.2ms pre-process, 17.0ms inference, 3.2ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5/runs/val/exp3
@ -708,14 +751,15 @@ model.
.. image:: 111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_33_0.png
.. image:: 111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_34_0.png
Inference Demo Performance Comparison
-------------------------------------
Inference Demo Performance Comparison `⇑ <#top>`__
###############################################################################################################################
This part shows how to use the Ultralytics model detection code
`“detect.py” <https://github.com/ultralytics/yolov5/blob/master/detect.py>`__
This part shows how to use the Ultralytics model detection code
`detect.py <https://github.com/ultralytics/yolov5/blob/master/detect.py>`__
to run synchronous inference, using the OpenVINO Python API on two
images.
@ -744,9 +788,9 @@ images.
'YOLOv5 🚀 v7.0-0-g915bbf2 Python-3.8.10 torch-1.13.1+cpu CPU',
'',
'Loading yolov5m/FP32_openvino_model for OpenVINO inference...',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 56.6ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 2 ties, 43.1ms',
'Speed: 1.4ms pre-process, 49.8ms inference, 1.2ms NMS per image at shape (1, 3, 640, 640)',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 57.0ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 2 ties, 40.6ms',
'Speed: 1.4ms pre-process, 48.8ms inference, 1.2ms NMS per image at shape (1, 3, 640, 640)',
'Results saved to \x1b[1mruns/detect/exp\x1b[0m']
@ -771,9 +815,9 @@ images.
'YOLOv5 🚀 v7.0-0-g915bbf2 Python-3.8.10 torch-1.13.1+cpu CPU',
'',
'Loading yolov5m/POT_INT8_openvino_model for OpenVINO inference...',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 38.2ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 1 tie, 33.4ms',
'Speed: 1.6ms pre-process, 35.8ms inference, 1.4ms NMS per image at shape (1, 3, 640, 640)',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 36.6ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 1 tie, 33.4ms',
'Speed: 1.5ms pre-process, 35.0ms inference, 1.4ms NMS per image at shape (1, 3, 640, 640)',
'Results saved to \x1b[1mruns/detect/exp2\x1b[0m']
@ -798,9 +842,9 @@ images.
'YOLOv5 🚀 v7.0-0-g915bbf2 Python-3.8.10 torch-1.13.1+cpu CPU',
'',
'Loading yolov5m/NNCF_INT8_openvino_model for OpenVINO inference...',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 37.5ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 2 ties, 32.4ms',
'Speed: 1.5ms pre-process, 35.0ms inference, 1.3ms NMS per image at shape (1, 3, 640, 640)',
'image 1/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, 35.9ms',
'image 2/2 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/111-yolov5-quantization-migration/yolov5/data/images/zidane.jpg: 640x640 3 persons, 2 ties, 31.5ms',
'Speed: 1.6ms pre-process, 33.7ms inference, 1.4ms NMS per image at shape (1, 3, 640, 640)',
'Results saved to \x1b[1mruns/detect/exp3\x1b[0m']
@ -828,11 +872,12 @@ images.
.. image:: 111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_39_0.png
.. image:: 111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_40_0.png
Benchmark
---------
Benchmark `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -864,7 +909,7 @@ Benchmark
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 34.56 ms
[ INFO ] Read model took 31.30 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,3,640,640]
@ -878,7 +923,7 @@ Benchmark
[ INFO ] Model outputs:
[ INFO ] output0 (node: output0) : f32 / [...] / [1,25200,85]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 382.88 ms
[ INFO ] Compile model took 360.18 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -900,17 +945,17 @@ Benchmark
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 106.76 ms
[ INFO ] First inference took 102.48 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 450 iterations
[ INFO ] Duration: 15298.04 ms
[ INFO ] Count: 456 iterations
[ INFO ] Duration: 15352.58 ms
[ INFO ] Latency:
[ INFO ] Median: 204.03 ms
[ INFO ] Average: 202.87 ms
[ INFO ] Min: 140.18 ms
[ INFO ] Max: 217.69 ms
[ INFO ] Throughput: 29.42 FPS
[ INFO ] Median: 202.33 ms
[ INFO ] Average: 201.47 ms
[ INFO ] Min: 138.12 ms
[ INFO ] Max: 216.53 ms
[ INFO ] Throughput: 29.70 FPS
.. code:: ipython3
@ -941,7 +986,7 @@ Benchmark
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 39.27 ms
[ INFO ] Read model took 44.00 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,3,640,640]
@ -955,7 +1000,7 @@ Benchmark
[ INFO ] Model outputs:
[ INFO ] output0 (node: output0) : f32 / [...] / [1,25200,85]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 409.09 ms
[ INFO ] Compile model took 387.13 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -977,17 +1022,17 @@ Benchmark
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 103.11 ms
[ INFO ] First inference took 101.31 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 456 iterations
[ INFO ] Duration: 15294.88 ms
[ INFO ] Duration: 15246.15 ms
[ INFO ] Latency:
[ INFO ] Median: 201.67 ms
[ INFO ] Average: 200.70 ms
[ INFO ] Min: 124.86 ms
[ INFO ] Max: 218.07 ms
[ INFO ] Throughput: 29.81 FPS
[ INFO ] Median: 200.30 ms
[ INFO ] Average: 199.86 ms
[ INFO ] Min: 96.50 ms
[ INFO ] Max: 219.99 ms
[ INFO ] Throughput: 29.91 FPS
.. code:: ipython3
@ -1018,7 +1063,7 @@ Benchmark
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 47.44 ms
[ INFO ] Read model took 45.57 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,3,640,640]
@ -1032,7 +1077,7 @@ Benchmark
[ INFO ] Model outputs:
[ INFO ] output0 (node: output0) : f32 / [...] / [1,25200,85]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 705.56 ms
[ INFO ] Compile model took 702.15 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -1054,17 +1099,17 @@ Benchmark
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 50.77 ms
[ INFO ] First inference took 48.39 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1422 iterations
[ INFO ] Duration: 15093.38 ms
[ INFO ] Count: 1410 iterations
[ INFO ] Duration: 15055.06 ms
[ INFO ] Latency:
[ INFO ] Median: 63.57 ms
[ INFO ] Average: 63.51 ms
[ INFO ] Min: 44.75 ms
[ INFO ] Max: 86.46 ms
[ INFO ] Throughput: 94.21 FPS
[ INFO ] Median: 64.05 ms
[ INFO ] Average: 63.87 ms
[ INFO ] Min: 46.43 ms
[ INFO ] Max: 83.05 ms
[ INFO ] Throughput: 93.66 FPS
.. code:: ipython3
@ -1095,7 +1140,7 @@ Benchmark
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 53.05 ms
[ INFO ] Read model took 52.31 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,3,640,640]
@ -1109,7 +1154,7 @@ Benchmark
[ INFO ] Model outputs:
[ INFO ] output0 (node: output0) : f32 / [...] / [1,25200,85]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 714.97 ms
[ INFO ] Compile model took 710.35 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -1131,26 +1176,27 @@ Benchmark
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 53.03 ms
[ INFO ] First inference took 51.01 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1422 iterations
[ INFO ] Duration: 15073.87 ms
[ INFO ] Count: 1416 iterations
[ INFO ] Duration: 15113.02 ms
[ INFO ] Latency:
[ INFO ] Median: 63.63 ms
[ INFO ] Average: 63.46 ms
[ INFO ] Min: 53.07 ms
[ INFO ] Max: 85.38 ms
[ INFO ] Throughput: 94.34 FPS
[ INFO ] Median: 63.94 ms
[ INFO ] Average: 63.87 ms
[ INFO ] Min: 45.06 ms
[ INFO ] Max: 87.95 ms
[ INFO ] Throughput: 93.69 FPS
References
----------
References `⇑ <#top>`__
###############################################################################################################################
- `Ultralytics YOLOv5 <https://github.com/ultralytics/yolov5>`__
- `OpenVINO Post-training Optimization
Tool <https://docs.openvino.ai/2023.0/pot_introduction.html>`__
- `NNCF Post-training
quantization <https://docs.openvino.ai/nightly/basic_quantization_flow.html>`__
- `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html#doxid-openvino-docs-m-o-d-g-deep-learning-model-optimizer-dev-guide>`__
quantization <https://docs.openvino.ai/2023.0/basic_quantization_flow.html>`__
- `Model Conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__

0
docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_33_0.png → docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_34_0.png
View File

0
docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_39_0.png → docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_40_0.png
View File

8
docs/notebooks/111-yolov5-quantization-migration-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="111-yolov5-quantization-migration-with-output_33_0.png">111-yolov5-quantization-migration-with-output_3..&gt;</a> 12-Jul-2023 00:11 33667
<a href="111-yolov5-quantization-migration-with-output_39_0.png">111-yolov5-quantization-migration-with-output_3..&gt;</a> 12-Jul-2023 00:11 770524
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="111-yolov5-quantization-migration-with-output_34_0.png">111-yolov5-quantization-migration-with-output_3..&gt;</a> 16-Aug-2023 01:31 33667
<a href="111-yolov5-quantization-migration-with-output_40_0.png">111-yolov5-quantization-migration-with-output_4..&gt;</a> 16-Aug-2023 01:31 770524
</pre><hr></body>
</html>

226
docs/notebooks/112-pytorch-post-training-quantization-nncf-with-output.rst
View File

@ -1,6 +1,8 @@
Post-Training Quantization of PyTorch models with NNCF
======================================================
.. _top:
The goal of this tutorial is to demonstrate how to use the NNCF (Neural
Network Compression Framework) 8-bit quantization in post-training mode
(without the fine-tuning pipeline) to optimize a PyTorch model for the
@ -20,10 +22,31 @@ quantization, not demanding the fine-tuning of the model.
**NOTE**: This notebook requires that a C++ compiler is accessible on
the default binary search path of the OS you are running the
notebook.
notebook.
**Table of contents**:
- `Preparations <#preparations>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Download and Prepare Tiny ImageNet dataset <#download-and-prepare-tiny-imagenet-dataset>`__
- `Helpers classes and functions <#helpers-classes-and-functions>`__
- `Validation function <#validation-function>`__
- `Create and load original uncompressed model <#create-and-load-original-uncompressed-model>`__
- `Create train and validation DataLoaders <#create-train-and-validation-dataloaders>`__
- `Model quantization and benchmarking <#model-quantization-and-benchmarking>`__
- `I. Evaluate the loaded model <#i-evaluate-the-loaded-model>`__
- `II. Create and initialize quantization <#ii-create-and-initialize-quantization>`__
- `III. Convert the models to OpenVINO Intermediate Representation (OpenVINO IR) <#iii-convert-the-models-to-openvino-intermediate-representation-openvino-ir>`__
- `IV. Compare performance of INT8 model and FP32 model in OpenVINO <#iv-compare-performance-of-int8-model-and-fp32-model-in-openvino>`__
Preparations `⇑ <#top>`__
###############################################################################################################################
Preparations
------------
.. code:: ipython3
@ -62,8 +85,9 @@ Preparations
os.environ["LIB"] = os.pathsep.join(b.library_dirs)
print(f"Added {vs_dir} to PATH")
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -88,10 +112,10 @@ Imports
.. parsed-literal::
2023-07-11 22:44:00.778229: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:44:00.812197: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:47:54.862445: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:47:54.896717: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:44:01.354577: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:47:55.440534: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -99,8 +123,9 @@ Imports
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
Settings
~~~~~~~~
Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -142,12 +167,13 @@ Settings
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/112-pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/112-pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
Download and Prepare Tiny ImageNet dataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Download and Prepare Tiny ImageNet dataset `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- 100k images of shape 3x64x64,
- 200 different classes: snake, spider, cat, truck, grasshopper, gull,
@ -206,11 +232,10 @@ Download and Prepare Tiny ImageNet dataset
Successfully downloaded and extracted dataset to: output
Helpers classes and functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Helpers classes and functions `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The code below will help to count accuracy and visualize validation
process.
The code below will help to count accuracy and visualize validation process.
.. code:: ipython3
@ -272,8 +297,9 @@ process.
return res
Validation function
~~~~~~~~~~~~~~~~~~~
Validation function `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -325,10 +351,11 @@ Validation function
)
return top1.avg
Create and load original uncompressed model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create and load original uncompressed model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ResNet-50 from the `torchivision
ResNet-50 from the ```torchivision``
repository <https://github.com/pytorch/vision>`__ is pre-trained on
ImageNet with more prediction classes than Tiny ImageNet, so the model
is adjusted by swapping the last FC layer to one with fewer output
@ -353,8 +380,9 @@ values.
model = create_model(MODEL_DIR / fp32_checkpoint_filename)
Create train and validation dataloaders
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create train and validation DataLoaders `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -403,15 +431,16 @@ Create train and validation dataloaders
train_loader, val_loader = create_dataloaders()
Model quantization and benchmarking
-----------------------------------
Model quantization and benchmarking `⇑ <#top>`__
###############################################################################################################################
With the validation pipeline, model files, and data-loading procedures
for model calibration now prepared, it’s time to proceed with the actual
post-training quantization using NNCF.
With the validation pipeline, model files, and data-loading procedures for model calibration
now prepared, it’s time to proceed with the actual post-training
quantization using NNCF.
I. Evaluate the loaded model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I. Evaluate the loaded model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -421,28 +450,28 @@ I. Evaluate the loaded model
.. parsed-literal::
Test: [ 0/79] Time 0.253 (0.253) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.238 (0.240) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.235 (0.239) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.234 (0.238) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.235 (0.238) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.232 (0.240) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.251 (0.240) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.237 (0.240) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 18.698
Test: [ 0/79] Time 0.240 (0.240) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.234 (0.227) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.220 (0.225) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.219 (0.223) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.225 (0.222) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.220 (0.222) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.219 (0.222) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.219 (0.222) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 17.387
Test accuracy of FP32 model: 60.740
II. Create and initialize quantization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
II. Create and initialize quantization `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
NNCF enables post-training quantization by adding the quantization
layers into the model graph and then using a subset of the training
dataset to initialize the parameters of these additional quantization
layers. The framework is designed so that modifications to your original
training code are minor. Quantization is the simplest scenario and
requires a few modifications. For more information about NNCF Post
Training Quantization (PTQ) API, refer to the `Basic Quantization Flow
NNCF enables post-training quantization by adding the quantization layers into the
model graph and then using a subset of the training dataset to
initialize the parameters of these additional quantization layers. The
framework is designed so that modifications to your original training
code are minor. Quantization is the simplest scenario and requires a few
modifications. For more information about NNCF Post Training
Quantization (PTQ) API, refer to the `Basic Quantization Flow
Guide <https://docs.openvino.ai/2023.0/basic_qauntization_flow.html#doxid-basic-qauntization-flow>`__.
1. Create a transformation function that accepts a sample from the
@ -498,16 +527,16 @@ Guide <https://docs.openvino.ai/2023.0/basic_qauntization_flow.html#doxid-basic-
.. parsed-literal::
Test: [ 0/79] Time 0.438 (0.438) Acc@1 81.25 (81.25) Acc@5 90.62 (90.62)
Test: [10/79] Time 0.428 (0.429) Acc@1 57.81 (67.26) Acc@5 86.72 (87.36)
Test: [20/79] Time 0.430 (0.429) Acc@1 67.97 (64.21) Acc@5 85.94 (87.43)
Test: [30/79] Time 0.429 (0.429) Acc@1 52.34 (62.45) Acc@5 75.78 (85.28)
Test: [40/79] Time 0.429 (0.429) Acc@1 68.75 (60.88) Acc@5 89.84 (84.47)
Test: [50/79] Time 0.429 (0.429) Acc@1 59.38 (60.81) Acc@5 87.50 (84.34)
Test: [60/79] Time 0.428 (0.429) Acc@1 66.41 (60.49) Acc@5 85.94 (83.68)
Test: [70/79] Time 0.434 (0.429) Acc@1 54.69 (60.27) Acc@5 78.91 (83.24)
* Acc@1 60.810 Acc@5 83.880 Total time: 33.587
Accuracy of initialized INT8 model: 60.810
Test: [ 0/79] Time 0.417 (0.417) Acc@1 80.47 (80.47) Acc@5 91.41 (91.41)
Test: [10/79] Time 0.413 (0.414) Acc@1 57.81 (66.76) Acc@5 85.94 (87.71)
Test: [20/79] Time 0.416 (0.413) Acc@1 66.41 (63.88) Acc@5 84.38 (87.65)
Test: [30/79] Time 0.414 (0.413) Acc@1 53.91 (62.17) Acc@5 76.56 (85.28)
Test: [40/79] Time 0.413 (0.413) Acc@1 67.97 (60.88) Acc@5 89.06 (84.41)
Test: [50/79] Time 0.414 (0.413) Acc@1 63.28 (60.86) Acc@5 87.50 (84.34)
Test: [60/79] Time 0.415 (0.413) Acc@1 65.62 (60.41) Acc@5 85.16 (83.67)
Test: [70/79] Time 0.413 (0.413) Acc@1 53.12 (60.15) Acc@5 79.69 (83.29)
* Acc@1 60.610 Acc@5 83.880 Total time: 32.356
Accuracy of initialized INT8 model: 60.610
It should be noted that the inference time for the quantized PyTorch
@ -516,18 +545,17 @@ added to the model by NNCF. However, the model’s performance will
significantly improve when it is in the OpenVINO Intermediate
Representation (IR) format.
III. Convert the models to OpenVINO Intermediate Representation (OpenVINO IR)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III. Convert the models to OpenVINO Intermediate Representation (OpenVINO IR) `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use Model Optimizer Python API to convert the Pytorch models to OpenVINO
IR. The models will be saved to the ‘OUTPUT’ directory for latter
benchmarking.
To convert the Pytorch models to OpenVINO IR, use model conversion
Python API . The models will be saved to the ‘OUTPUT’ directory for
later benchmarking.
For more information about Model Optimizer, refer to the `Model
Optimizer Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
For more information about model conversion, refer to this
`page <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
Before converting models export them to ONNX. Executing the following
Before converting models, export them to ONNX. Executing the following
command may take a while.
.. code:: ipython3
@ -549,17 +577,17 @@ command may take a while.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:338: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:338: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:346: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:346: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/quantize_functions.py:140: FutureWarning: 'torch.onnx._patch_torch._graph_op' is deprecated in version 1.13 and will be removed in version 1.14. Please note 'g.op()' is to be removed from torch.Graph. Please open a GitHub issue if you need this functionality..
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/quantize_functions.py:140: FutureWarning: 'torch.onnx._patch_torch._graph_op' is deprecated in version 1.13 and will be removed in version 1.14. Please note 'g.op()' is to be removed from torch.Graph. Please open a GitHub issue if you need this functionality..
output = g.op(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_patch_torch.py:81: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_patch_torch.py:81: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_node_shape_type_inference(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of org.openvinotoolkit::FakeQuantize type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
@ -600,15 +628,15 @@ Evaluate the FP32 and INT8 models.
.. parsed-literal::
Test: [ 0/79] Time 0.205 (0.205) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.137 (0.143) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.145 (0.141) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.137 (0.140) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.137 (0.140) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.139 (0.140) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.135 (0.139) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.139 (0.139) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 10.882
Test: [ 0/79] Time 0.200 (0.200) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.138 (0.144) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.137 (0.141) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.136 (0.140) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.139 (0.140) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.135 (0.139) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.139 (0.139) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.138 (0.139) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 10.865
Accuracy of FP32 IR model: 60.740
@ -621,20 +649,20 @@ Evaluate the FP32 and INT8 models.
.. parsed-literal::
Test: [ 0/79] Time 0.187 (0.187) Acc@1 81.25 (81.25) Acc@5 90.62 (90.62)
Test: [10/79] Time 0.082 (0.091) Acc@1 57.81 (66.90) Acc@5 85.94 (87.78)
Test: [20/79] Time 0.080 (0.086) Acc@1 67.97 (64.06) Acc@5 83.59 (87.31)
Test: [30/79] Time 0.081 (0.084) Acc@1 52.34 (62.17) Acc@5 78.12 (85.36)
Test: [40/79] Time 0.079 (0.083) Acc@1 67.97 (60.80) Acc@5 89.84 (84.38)
Test: [50/79] Time 0.077 (0.082) Acc@1 60.16 (60.71) Acc@5 87.50 (84.30)
Test: [60/79] Time 0.079 (0.082) Acc@1 67.19 (60.43) Acc@5 87.50 (83.72)
Test: [70/79] Time 0.082 (0.081) Acc@1 53.12 (60.17) Acc@5 80.47 (83.31)
* Acc@1 60.730 Acc@5 83.930 Total time: 6.353
Accuracy of INT8 IR model: 60.730
Test: [ 0/79] Time 0.189 (0.189) Acc@1 81.25 (81.25) Acc@5 91.41 (91.41)
Test: [10/79] Time 0.079 (0.091) Acc@1 59.38 (66.90) Acc@5 85.94 (87.43)
Test: [20/79] Time 0.078 (0.087) Acc@1 67.19 (64.25) Acc@5 85.16 (87.28)
Test: [30/79] Time 0.080 (0.085) Acc@1 51.56 (62.40) Acc@5 75.78 (85.21)
Test: [40/79] Time 0.077 (0.083) Acc@1 67.97 (60.94) Acc@5 89.84 (84.51)
Test: [50/79] Time 0.078 (0.082) Acc@1 62.50 (61.06) Acc@5 87.50 (84.45)
Test: [60/79] Time 0.081 (0.082) Acc@1 66.41 (60.71) Acc@5 85.94 (83.84)
Test: [70/79] Time 0.078 (0.082) Acc@1 52.34 (60.40) Acc@5 79.69 (83.42)
* Acc@1 60.930 Acc@5 84.020 Total time: 6.371
Accuracy of INT8 IR model: 60.930
IV. Compare performance of INT8 model and FP32 model in OpenVINO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IV. Compare performance of INT8 model and FP32 model in OpenVINO `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Finally, measure the inference performance of the ``FP32`` and ``INT8``
models, using `Benchmark
@ -693,13 +721,13 @@ throughput (frames per second) values.
.. parsed-literal::
Benchmark FP32 model (OpenVINO IR)
[ INFO ] Throughput: 37.61 FPS
[ INFO ] Throughput: 37.93 FPS
Benchmark INT8 model (OpenVINO IR)
[ INFO ] Throughput: 159.33 FPS
[ INFO ] Throughput: 155.44 FPS
Benchmark FP32 model (OpenVINO IR) synchronously
[ INFO ] Throughput: 39.06 FPS
[ INFO ] Throughput: 38.81 FPS
Benchmark INT8 model (OpenVINO IR) synchronously
[ INFO ] Throughput: 140.24 FPS
[ INFO ] Throughput: 139.97 FPS
Show device Information for reference:

164
docs/notebooks/113-image-classification-quantization-with-output.rst
View File

@ -1,10 +1,12 @@
Quantization of Image Classification Models
===========================================
.. _top:
This tutorial demonstrates how to apply ``INT8`` quantization to Image
Classification model using
`NNCF <https://github.com/openvinotoolkit/nncf>`__. It uses the
Mobilenet V2 model, trained on Cifar10 dataset. The code is designed to
MobileNet V2 model, trained on Cifar10 dataset. The code is designed to
be extendable to custom models and datasets. The tutorial uses OpenVINO
backend for performing model quantization in NNCF, if you interested how
to apply quantization on PyTorch model, please check this
@ -12,12 +14,29 @@ to apply quantization on PyTorch model, please check this
This tutorial consists of the following steps:
- Prepare the model for quantization.
- Define a data loading functionality.
- Perform quantization.
- Compare accuracy of the original and quantized models.
- Compare performance of the original and quantized models.
- Compare results on one picture.
- Prepare the model for quantization.
- Define a data loading functionality.
- Perform quantization.
- Compare accuracy of the original and quantized models.
- Compare performance of the original and quantized models.
- Compare results on one picture.
**Table of contents**:
- `Prepare the Model <#prepare-the-model>`__
- `Prepare Dataset <#prepare-dataset>`__
- `Perform Quantization <#perform-quantization>`__
- `Create Dataset for Validation <#create-dataset-for-validation>`__
- `Run nncf.quantize for Getting an Optimized Model <#run-nncf.quantize-for-getting-an-optimized-model>`__
- `Serialize an OpenVINO IR model <#serialize-an-openvino-ir-model>`__
- `Compare Accuracy of the Original and Quantized Models <#compare-accuracy-of-the-original-and-quantized-models>`__
- `Select inference device <#select-inference-device>`__
- `Compare Performance of the Original and Quantized Models <#compare-performance-of-the-original-and-quantized-models>`__
- `Compare results on four pictures <#compare-results-on-four-pictures>`__
.. code:: ipython3
@ -31,14 +50,16 @@ This tutorial consists of the following steps:
DATA_DIR.mkdir(exist_ok=True)
MODEL_DIR.mkdir(exist_ok=True)
Prepare the Model
-----------------
Prepare the Model `⇑ <#top>`__
###############################################################################################################################
Model preparation stage has the following steps:
- Download a PyTorch model
- Convert model to OpenVINO Intermediate Representation format (IR) using Model Optimizer Python API
- Serialize converted model on disk
- Download a PyTorch model
- Convert model to OpenVINO Intermediate Representation format (IR)
using model conversion Python API
- Serialize converted model on disk
.. code:: ipython3
@ -57,7 +78,7 @@ Model preparation stage has the following steps:
remote: Counting objects: 100% (281/281), done.
remote: Compressing objects: 100% (95/95), done.
remote: Total 282 (delta 136), reused 269 (delta 129), pack-reused 1
Receiving objects: 100% (282/282), 9.22 MiB | 4.20 MiB/s, done.
Receiving objects: 100% (282/282), 9.22 MiB | 4.67 MiB/s, done.
Resolving deltas: 100% (136/136), done.
@ -67,17 +88,17 @@ Model preparation stage has the following steps:
model = cifar10_mobilenetv2_x1_0(pretrained=True)
OpenVINO support PyTorch models via conversion to OpenVINO Intermediate
Representation format. Model Optimizer Python API should be used for
conversion. ``mo.convert_model`` accept PyTorch model instance and
convert it into ``openvino.runtime.Model`` representation of model in
OpenVINO. Optionally, you may specify ``example_input`` which serves as
helper for model tracing and ``input_shape`` for conversion model with
static shape. Conveted model is ready to be loaded on device for
inference and can be saved on disk for next usage via ``serialize``
function. More details about Model Optimizer Python API can be found on
this
`page <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__.
OpenVINO supports PyTorch models via conversion to OpenVINO Intermediate
Representation format using model conversion Python API.
``mo.convert_model`` accept PyTorch model instance and convert it into
``openvino.runtime.Model`` representation of model in OpenVINO.
Optionally, you may specify ``example_input`` which serves as a helper
for model tracing and ``input_shape`` for converting the model with
static shape. The converted model is ready to be loaded on a device for
inference and can be saved on a disk for next usage via the
``serialize`` function. More details about model conversion Python API
can be found on this
`page <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
.. code:: ipython3
@ -90,8 +111,9 @@ this
serialize(ov_model, MODEL_DIR / "mobilenet_v2.xml")
Prepare Dataset
---------------
Prepare Dataset `⇑ <#top>`__
###############################################################################################################################
We will use `CIFAR10 <https://www.cs.toronto.edu/~kriz/cifar.html>`__
dataset from
@ -132,22 +154,24 @@ Preprocessing for model obtained from training
Extracting ../data/datasets/cifar10/cifar-10-python.tar.gz to ../data/datasets/cifar10
Perform Quantization
--------------------
Perform Quantization `⇑ <#top>`__
###############################################################################################################################
`NNCF <https://github.com/openvinotoolkit/nncf>`__ provides a suite of
advanced algorithms for Neural Networks inference optimization in
OpenVINO with minimal accuracy drop. We will use 8-bit quantization in
post-training mode (without the fine-tuning pipeline) to optimize
MobilenetV2. The optimization process contains the following steps:
MobileNetV2. The optimization process contains the following steps:
1. Create a Dataset for quantization.
2. Run ``nncf.quantize`` for getting an optimized model.
3. Serialize an OpenVINO IR model, using the
``openvino.runtime.serialize`` function.
Create Datset for Validation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create Dataset for Validation `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
NNCF is compatible with ``torch.utils.data.DataLoader`` interface. For
performing quantization it should be passed into ``nncf.Dataset`` object
@ -171,8 +195,9 @@ model during quantization, in our case, to pick input tensor from pair
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
Run nncf.quantize for Getting an Optimized Model
------------------------------------------------
Run nncf.quantize for Getting an Optimized Model `⇑ <#top>`__
###############################################################################################################################
``nncf.quantize`` function accepts model and prepared quantization
dataset for performing basic quantization. Optionally, additional
@ -188,12 +213,13 @@ about supported parameters can be found on this
.. parsed-literal::
Statistics collection: 100%|██████████| 300/300 [00:08<00:00, 35.03it/s]
Biases correction: 100%|██████████| 36/36 [00:01<00:00, 23.98it/s]
Statistics collection: 100%|██████████| 300/300 [00:08<00:00, 35.62it/s]
Biases correction: 100%|██████████| 36/36 [00:01<00:00, 19.40it/s]
Serialize an OpenVINO IR model
------------------------------
Serialize an OpenVINO IR model `⇑ <#top>`__
###############################################################################################################################
Similar to ``mo.convert_model``, quantized model is
``openvino.runtime.Model`` object which ready to be loaded into device
@ -203,8 +229,9 @@ and can be serialized on disk using ``openvino.runtime.serialize``.
serialize(quant_ov_model, MODEL_DIR / "quantized_mobilenet_v2.xml")
Compare Accuracy of the Original and Quantized Models
-----------------------------------------------------
Compare Accuracy of the Original and Quantized Models `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -221,10 +248,11 @@ Compare Accuracy of the Original and Quantized Models
total += 1
return correct / total
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -287,8 +315,9 @@ select device from dropdown list for running inference using OpenVINO
Accuracy of the optimized model: 93.51%
Compare Performance of the Original and Quantized Models
--------------------------------------------------------
Compare Performance of the Original and Quantized Models `⇑ <#top>`__
###############################################################################################################################
Finally, measure the inference performance of the ``FP32`` and ``INT8``
models, using `Benchmark
@ -328,18 +357,18 @@ Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_
[ INFO ] Read model took 7.88 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x.1 , x , 1 (node: Parameter_2) : f32 / [...] / [1,3,32,32]
[ INFO ] x.1 , 1 , x (node: Parameter_2) : f32 / [...] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ] 223 (node: aten::linear_928) : f32 / [...] / [1,10]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x.1 , x , 1 (node: Parameter_2) : u8 / [N,C,H,W] / [1,3,32,32]
[ INFO ] x.1 , 1 , x (node: Parameter_2) : u8 / [N,C,H,W] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ] 223 (node: aten::linear_928) : f32 / [...] / [1,10]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 164.70 ms
[ INFO ] Compile model took 186.19 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
@ -368,17 +397,17 @@ Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_
[ INFO ] Fill input '1' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 3.15 ms
[ INFO ] First inference took 3.04 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 89880 iterations
[ INFO ] Duration: 15002.85 ms
[ INFO ] Count: 89724 iterations
[ INFO ] Duration: 15003.40 ms
[ INFO ] Latency:
[ INFO ] Median: 1.80 ms
[ INFO ] Average: 1.82 ms
[ INFO ] Min: 1.13 ms
[ INFO ] Max: 7.45 ms
[ INFO ] Throughput: 5990.86 FPS
[ INFO ] Min: 1.29 ms
[ INFO ] Max: 9.73 ms
[ INFO ] Throughput: 5980.25 FPS
.. code:: ipython3
@ -404,21 +433,21 @@ Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 15.06 ms
[ INFO ] Read model took 14.79 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] 1 , x.1 , x (node: Parameter_2) : f32 / [...] / [1,3,32,32]
[ INFO ] x , 1 , x.1 (node: Parameter_2) : f32 / [...] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ] 223 (node: aten::linear_928) : f32 / [...] / [1,10]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] 1 , x.1 , x (node: Parameter_2) : u8 / [N,C,H,W] / [1,3,32,32]
[ INFO ] x , 1 , x.1 (node: Parameter_2) : u8 / [N,C,H,W] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ] 223 (node: aten::linear_928) : f32 / [...] / [1,10]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 314.58 ms
[ INFO ] Compile model took 300.34 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
@ -447,21 +476,22 @@ Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_
[ INFO ] Fill input '1' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 1.86 ms
[ INFO ] First inference took 1.83 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 181056 iterations
[ INFO ] Duration: 15001.51 ms
[ INFO ] Count: 181212 iterations
[ INFO ] Duration: 15001.37 ms
[ INFO ] Latency:
[ INFO ] Median: 0.93 ms
[ INFO ] Average: 0.96 ms
[ INFO ] Min: 0.62 ms
[ INFO ] Max: 3.36 ms
[ INFO ] Throughput: 12069.19 FPS
[ INFO ] Average: 0.95 ms
[ INFO ] Min: 0.63 ms
[ INFO ] Max: 6.70 ms
[ INFO ] Throughput: 12079.70 FPS
Compare results on four pictures
--------------------------------
Compare results on four pictures `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -551,5 +581,5 @@ Compare results on four pictures
.. image:: 113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_28_2.png
.. image:: 113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_29_2.png

0
docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_28_2.png → docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_29_2.png
View File

6
docs/notebooks/113-image-classification-quantization-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/113-image-classification-quantization-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/113-image-classification-quantization-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/113-image-classification-quantization-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="113-image-classification-quantization-with-output_28_2.png">113-image-classification-quantization-with-outp..&gt;</a> 12-Jul-2023 00:11 14855
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/113-image-classification-quantization-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="113-image-classification-quantization-with-output_29_2.png">113-image-classification-quantization-with-outp..&gt;</a> 16-Aug-2023 01:31 14855
</pre><hr></body>
</html>

155
docs/notebooks/115-async-api-with-output.rst
View File

@ -1,8 +1,10 @@
Asynchronous Inference with OpenVINO™
=====================================
.. _top:
This notebook demonstrates how to use the `Async
API <https://docs.openvino.ai/nightly/openvino_docs_deployment_optimization_guide_common.html>`__
API <https://docs.openvino.ai/2023.0/openvino_docs_deployment_optimization_guide_common.html>`__
for asynchronous execution with OpenVINO.
OpenVINO Runtime supports inference in either synchronous or
@ -11,12 +13,37 @@ device is busy with inference, the application can perform other tasks
in parallel (for example, populating inputs or scheduling other
requests) rather than wait for the current inference to complete first.
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Prepare model and data processing <#prepare-model-and-data-processing>`__
- `Download test model <#download-test-model>`__
- `Load the model <#load-the-model>`__
- `Create functions for data processing <#create-functions-for-data-processing>`__
- `Get the test video <#get-the-test-video>`__
- `How to improve the throughput of video processing <#how-to-improve-the-throughput-of-video-processing>`__
- `Sync Mode (default) <#sync-mode-default>`__
- `Test performance in Sync Mode <#test-performance-in-sync-mode>`__
- `Async Mode <#async-mode>`__
- `Test the performance in Async Mode <#test-the-performance-in-async-mode>`__
- `Compare the performance <#compare-the-performance>`__
- `AsyncInferQueue <#asyncinferqueue>`__
- `Setting Callback <#setting-callback>`__
- `Test the performance with AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
!pip install -q 'openvino-dev>=2023.0.0'
!pip install -q "openvino-dev>=2023.0.0"
!pip install -q opencv-python matplotlib
.. code:: ipython3
@ -38,14 +65,15 @@ Imports
import notebook_utils as utils
Prepare model and data processing
---------------------------------
Prepare model and data processing `⇑ <#top>`__
###############################################################################################################################
Download test model
~~~~~~~~~~~~~~~~~~~
We use a pre-trained model from OpenVINO’s `Open Model
Zoo <https://docs.openvino.ai/nightly/model_zoo.html>`__ to start the
Download test model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
We use a pre-trained model from OpenVINO’s
`Open Model Zoo <https://docs.openvino.ai/2023.0/model_zoo.html>`__ to start the
test. In this case, the model will be executed to detect the person in
each frame of the video.
@ -80,8 +108,9 @@ each frame of the video.
Load the model
~~~~~~~~~~~~~~
Load the model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -91,7 +120,7 @@ Load the model
# read the network and corresponding weights from file
model = ie.read_model(model=model_path)
# compile the model for the CPU (you can choose manually CPU, GPU, etc.)
# compile the model for the CPU (you can choose manually CPU, GPU etc.)
# or let the engine choose the best available device (AUTO)
compiled_model = ie.compile_model(model=model, device_name="CPU")
@ -100,8 +129,9 @@ Load the model
N, C, H, W = input_layer_ir.shape
shape = (H, W)
Create functions for data processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create functions for data processing `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -142,27 +172,30 @@ Create functions for data processing
cv2.putText(image, str(round(fps, 2)) + " fps", (5, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 3)
return image
Get the test video
~~~~~~~~~~~~~~~~~~
Get the test video `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
video_path = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/CEO%20Pat%20Gelsinger%20on%20Leading%20Intel.mp4'
How to improve the throughput of video processing
-------------------------------------------------
How to improve the throughput of video processing `⇑ <#top>`__
###############################################################################################################################
Below, we compare the performance of the synchronous and async-based
approaches:
Sync Mode (default)
~~~~~~~~~~~~~~~~~~~
Sync Mode (default) `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Let us see how video processing works with the default approach. Using
the synchronous approach, the frame is captured with OpenCV and then
immediately processed:
Let us see how video processing works with the default approach. Using the synchronous approach, the frame is
captured with OpenCV and then immediately processed:
.. image:: https://camo.githubusercontent.com/b77ae49e3c46a0fe3931ad02a7767d6def49aa7b2a7a8ad620a4e12ae31d43a5/68747470733a2f2f757365722d696d616765732e67697468756275736572636f6e74656e742e636f6d2f39313233373932342f3136383435323537332d64333534656135622d373936362d343465352d383133642d6639303533626534333338612e706e67
.. figure:: https://user-images.githubusercontent.com/91237924/168452573-d354ea5b-7966-44e5-813d-f9053be4338a.png
:alt: drawing
drawing
::
@ -174,6 +207,8 @@ immediately processed:
// display CURRENT result
}
\``\`
.. code:: ipython3
def sync_api(source, flip, fps, use_popup, skip_first_frames):
@ -240,8 +275,9 @@ immediately processed:
player.stop()
return sync_fps
Test performance in Sync Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Test performance in Sync Mode `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -250,15 +286,18 @@ Test performance in Sync Mode
.. image:: 115-async-api-with-output_files/115-async-api-with-output_15_0.png
.. parsed-literal::
Source ended
average throuput in sync mode: 38.25 fps
average throuput in sync mode: 37.71 fps
Async Mode
~~~~~~~~~~
Async Mode `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Let us see how the OpenVINO Async API can improve the overall frame rate
of an application. The key advantage of the Async approach is as
@ -267,7 +306,10 @@ do other things in parallel (for example, populating inputs or
scheduling other requests) rather than wait for the current inference to
complete first.
.. image:: https://camo.githubusercontent.com/7bcadb7cf72aefc84d74d8e971a31dab1710bb4303fef814619df638dff8be92/68747470733a2f2f757365722d696d616765732e67697468756275736572636f6e74656e742e636f6d2f39313233373932342f3136383435323537322d63326666316335392d643437302d346238352d623166362d6236653164616339353430652e706e67
.. figure:: https://user-images.githubusercontent.com/91237924/168452572-c2ff1c59-d470-4b85-b1f6-b6e1dac9540e.png
:alt: drawing
drawing
In the example below, inference is applied to the results of the video
decoding. So it is possible to keep multiple infer requests, and while
@ -369,8 +411,9 @@ pipeline (decoding vs inference) and not by the sum of the stages.
player.stop()
return async_fps
Test the performance in Async Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Test the performance in Async Mode `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -378,14 +421,19 @@ Test the performance in Async Mode
print(f"average throuput in async mode: {async_fps:.2f} fps")
.. image:: 115-async-api-with-output_files/115-async-api-with-output_19_0.png
.. parsed-literal::
Source ended
average throuput in async mode: 71.88 fps
average throuput in async mode: 73.36 fps
Compare the performance
~~~~~~~~~~~~~~~~~~~~~~~
Compare the performance `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -409,24 +457,29 @@ Compare the performance
AsyncInferQueue
---------------
.. image:: 115-async-api-with-output_files/115-async-api-with-output_21_0.png
``AsyncInferQueue`` `⇑ <#top>`__
###############################################################################################################################
Asynchronous mode pipelines can be supported with the
`AsyncInferQueue <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_Python_API_exclusives.html#asyncinferqueue>`__
wrapper class. This class automatically spawns the pool of InferRequest
objects (also called “jobs”) and provides synchronization mechanisms to
control the flow of the pipeline. It is a simpler way to manage the
infer request queue in Asynchronous mode.
wrapper class. This class automatically spawns the pool of
``InferRequest`` objects (also called “jobs”) and provides
synchronization mechanisms to control the flow of the pipeline. It is a
simpler way to manage the infer request queue in Asynchronous mode.
Setting Callback `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Setting Callback
~~~~~~~~~~~~~~~~
When ``callback`` is set, any job that ends inference calls upon the
Python function. The ``callback`` function must have two arguments: one
is the request that calls the ``callback``, which provides the
InferRequest API; the other is called “userdata”, which provides the
possibility of passing runtime values.
``InferRequest`` API; the other is called “user data”, which provides
the possibility of passing runtime values.
.. code:: ipython3
@ -496,8 +549,9 @@ possibility of passing runtime values.
infer_queue.wait_all()
player.stop()
Test the performance with AsyncInferQueue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Test the performance with ``AsyncInferQueue`` `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -508,7 +562,10 @@ Test the performance with AsyncInferQueue
.. image:: 115-async-api-with-output_files/115-async-api-with-output_27_0.png
.. parsed-literal::
average throughput in async mode with async infer queue: 104.69 fps
average throughput in async mode with async infer queue: 103.73 fps

3
docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b94a4331de267d1151abf422719316d87b59282125c1cb8471a99e34d561bdd1
size 30455

12
docs/notebooks/115-async-api-with-output_files/index.html
View File

@ -1,10 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/115-async-api-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/115-async-api-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/115-async-api-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="115-async-api-with-output_15_0.png">115-async-api-with-output_15_0.png</a> 12-Jul-2023 00:11 4307
<a href="115-async-api-with-output_19_0.png">115-async-api-with-output_19_0.png</a> 12-Jul-2023 00:11 4307
<a href="115-async-api-with-output_21_0.png">115-async-api-with-output_21_0.png</a> 12-Jul-2023 00:11 30421
<a href="115-async-api-with-output_27_0.png">115-async-api-with-output_27_0.png</a> 12-Jul-2023 00:11 4307
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/115-async-api-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="115-async-api-with-output_15_0.png">115-async-api-with-output_15_0.png</a> 16-Aug-2023 01:31 4307
<a href="115-async-api-with-output_19_0.png">115-async-api-with-output_19_0.png</a> 16-Aug-2023 01:31 4307
<a href="115-async-api-with-output_21_0.png">115-async-api-with-output_21_0.png</a> 16-Aug-2023 01:31 30455
<a href="115-async-api-with-output_27_0.png">115-async-api-with-output_27_0.png</a> 16-Aug-2023 01:31 4307
</pre><hr></body>
</html>

109
docs/notebooks/116-sparsity-optimization-with-output.rst
View File

@ -1,14 +1,14 @@
Accelerate Inference of Sparse Transformer Models with OpenVINO™ and 4th Gen Intel® Xeon® Scalable Processors
=============================================================================================================
.. _top:
This tutorial demonstrates how to improve performance of sparse
Transformer models with `OpenVINO <https://docs.openvino.ai/>`__ on 4th
Gen Intel® Xeon® Scalable processors.
The tutorial downloads `a BERT-base
model <https://huggingface.co/OpenVINO/bert-base-uncased-sst2-int8-unstructured80>`__
which has been quantized, sparsified, and tuned for `SST2
datasets <https://huggingface.co/datasets/sst2>`__ using
The tutorial downloads `a BERT-base model <https://huggingface.co/OpenVINO/bert-base-uncased-sst2-int8-unstructured80>`__
which has been quantized, sparsified, and tuned for `SST2 datasets <https://huggingface.co/datasets/sst2>`__ using
`Optimum-Intel <https://github.com/huggingface/optimum-intel>`__. It
demonstrates the inference performance advantage on 4th Gen Intel® Xeon®
Scalable Processors by running it with `Sparse Weight
@ -21,16 +21,29 @@ consists of the following steps:
integration with Hugging Face Optimum.
- Compare sparse 8-bit vs. dense 8-bit inference performance.
Prerequisites
-------------
**Table of contents**:
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
- `Download, quantize and sparsify the model, using Hugging Face Optimum API <#download-quantize-and-sparsify-the-model-using-hugging-face-optimum-api>`__
- `Benchmark quantized dense inference performance <#benchmark-quantized-dense-inference-performance>`__
- `Benchmark quantized sparse inference performance <#benchmark-quantized-sparse-inference-performance>`__
- `When this might be helpful <#when-this-might-be-helpful>`__
Prerequisites `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
!pip install -q 'openvino-dev>=2023.0.0'
!pip install -q "openvino-dev>=2023.0.0"
!pip install -q "git+https://github.com/huggingface/optimum-intel.git" datasets onnx onnxruntime
Imports
-------
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -44,10 +57,10 @@ Imports
.. parsed-literal::
2023-07-11 22:51:18.663091: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:51:18.697477: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:55:04.775263: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:55:04.809127: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:51:19.248196: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:55:05.351203: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -60,8 +73,8 @@ Imports
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Download, quantize and sparsify the model, using Hugging Face Optimum API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Download, quantize and sparsify the model, using Hugging Face Optimum API. `⇑ <#top>`__
###############################################################################################################################
The first step is to download a quantized sparse transformers which has
been translated to OpenVINO IR. Then, it will be put through a
@ -128,14 +141,14 @@ the IRs into a single folder.
Benchmark quantized dense inference performance
-----------------------------------------------
Benchmark quantized dense inference performance `⇑ <#top>`__
###############################################################################################################################
Benchmark dense inference performance using parallel execution on four
CPU cores to simulate a small instance in the cloud infrastructure.
Sequence length is dependent on use cases, 16 is common for
conversational AI while 160 for question answering task. It is set to 64
as an example. It is recommended to tune based on your applications.
Benchmark dense inference performance using parallel execution on four CPU cores
to simulate a small instance in the cloud infrastructure. Sequence
length is dependent on use cases, 16 is common for conversational AI
while 160 for question answering task. It is set to 64 as an example. It
is recommended to tune based on your applications.
.. code:: ipython3
@ -175,7 +188,7 @@ as an example. It is recommended to tune based on your applications.
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 79.41 ms
[ INFO ] Read model took 74.55 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@ -186,7 +199,7 @@ as an example. It is recommended to tune based on your applications.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
[ INFO ] Reshape model took 26.38 ms
[ INFO ] Reshape model took 26.03 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@ -195,7 +208,7 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Model outputs:
[ INFO ] logits (node: logits) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1252.95 ms
[ INFO ] Compile model took 1231.43 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -221,21 +234,22 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 26.89 ms
[ INFO ] First inference took 31.02 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 8944 iterations
[ INFO ] Duration: 60031.11 ms
[ INFO ] Count: 8896 iterations
[ INFO ] Duration: 60044.19 ms
[ INFO ] Latency:
[ INFO ] Median: 26.64 ms
[ INFO ] Average: 26.68 ms
[ INFO ] Min: 25.23 ms
[ INFO ] Max: 40.15 ms
[ INFO ] Throughput: 148.99 FPS
[ INFO ] Median: 26.82 ms
[ INFO ] Average: 26.87 ms
[ INFO ] Min: 25.13 ms
[ INFO ] Max: 38.04 ms
[ INFO ] Throughput: 148.16 FPS
Benchmark quantized sparse inference performance
------------------------------------------------
Benchmark quantized sparse inference performance `⇑ <#top>`__
###############################################################################################################################
To enable sparse weight decompression feature, users can add it to
runtime config like below. ``CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE``
@ -282,7 +296,7 @@ for which a layer will be enabled.
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 69.37 ms
[ INFO ] Read model took 63.96 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@ -293,7 +307,7 @@ for which a layer will be enabled.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
[ INFO ] Reshape model took 25.96 ms
[ INFO ] Reshape model took 26.17 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@ -302,7 +316,7 @@ for which a layer will be enabled.
[ INFO ] Model outputs:
[ INFO ] logits (node: logits) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1250.91 ms
[ INFO ] Compile model took 1252.94 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -328,21 +342,22 @@ for which a layer will be enabled.
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 32.79 ms
[ INFO ] First inference took 30.12 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 8972 iterations
[ INFO ] Duration: 60029.38 ms
[ INFO ] Count: 8840 iterations
[ INFO ] Duration: 60036.27 ms
[ INFO ] Latency:
[ INFO ] Median: 26.59 ms
[ INFO ] Average: 26.63 ms
[ INFO ] Min: 25.42 ms
[ INFO ] Max: 41.12 ms
[ INFO ] Throughput: 149.46 FPS
[ INFO ] Median: 26.83 ms
[ INFO ] Average: 26.88 ms
[ INFO ] Min: 26.11 ms
[ INFO ] Max: 40.45 ms
[ INFO ] Throughput: 147.24 FPS
When this might be helpful
--------------------------
When this might be helpful `⇑ <#top>`__
###############################################################################################################################
This feature can improve inference performance for models with sparse
weights in the scenarios when the model is deployed to handle multiple

159
docs/notebooks/117-model-server-with-output.rst
View File

@ -1,6 +1,8 @@
Hello Model Server
==================
.. _top:
Introduction to OpenVINO™ Model Server (OVMS).
What is Model Serving?
@ -12,37 +14,54 @@ the model server, which performs inference and sends a response back to
the client. Model serving offers many advantages for efficient model
deployment:
- Remote inference enables using lightweight clients with only the
necessary functions to perform API calls to edge or cloud
deployments.
- Applications are independent of the model framework, hardware device,
and infrastructure.
- Client applications in any programming language that supports REST or
gRPC calls can be used to run inference remotely on the model server.
- Clients require fewer updates since client libraries change very
rarely.
- Model topology and weights are not exposed directly to client
applications, making it easier to control access to the model.
- Ideal architecture for microservices-based applications and
deployments in cloud environments – including Kubernetes and
OpenShift clusters.
- Efficient resource utilization with horizontal and vertical inference
scaling.
- Remote inference enables using lightweight clients with only the
necessary functions to perform API calls to edge or cloud
deployments.
- Applications are independent of the model framework, hardware device,
and infrastructure.
- Client applications in any programming language that supports REST or
gRPC calls can be used to run inference remotely on the model server.
- Clients require fewer updates since client libraries change very
rarely.
- Model topology and weights are not exposed directly to client
applications, making it easier to control access to the model.
- Ideal architecture for microservices-based applications and
deployments in cloud environments – including Kubernetes and
OpenShift clusters.
- Efficient resource utilization with horizontal and vertical inference
scaling.
.. figure:: https://user-images.githubusercontent.com/91237924/215658773-4720df00-3b95-4a84-85a2-40f06138e914.png
:alt: ovms_diagram
|ovms_diagram|
ovms_diagram
**Table of contents**:
Serving with OpenVINO Model Server
----------------------------------
- `Serving with OpenVINO Model Server <#serving-with-openvino-model-server1>`__
- `Step 1: Prepare Docker <#step-1-prepare-docker>`__
- `Step 2: Preparing a Model Repository <#step-2-preparing-a-model-repository>`__
- `Step 3: Start the Model Server Container <#start-the-model-server-container>`__
- `Step 4: Prepare the Example Client Components <#prepare-the-example-client-components>`__
OpenVINO Model Server (OVMS) is a high-performance system for serving
models. Implemented in C++ for scalability and optimized for deployment
on Intel architectures, the model server uses the same architecture and
API as TensorFlow Serving and KServe while applying OpenVINO for
inference execution. Inference service is provided via gRPC or REST API,
making deploying new algorithms and AI experiments easy.
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
- `Request Model Status <#request-model-status>`__
- `Request Model Metadata <#request-model-metadata>`__
- `Load input image <#load-input-image>`__
- `Request Prediction on a Numpy Array <#request-prediction-on-a-numpy-array>`__
- `Visualization <#visualization>`__
- `References <#references>`__
.. |ovms_diagram| image:: https://user-images.githubusercontent.com/91237924/215658773-4720df00-3b95-4a84-85a2-40f06138e914.png
Serving with OpenVINO Model Server `⇑ <#top>`__
###############################################################################################################################
OpenVINO Model Server (OVMS) is a high-performance system for serving models. Implemented in
C++ for scalability and optimized for deployment on Intel architectures,
the model server uses the same architecture and API as TensorFlow
Serving and KServe while applying OpenVINO for inference execution.
Inference service is provided via gRPC or REST API, making deploying new
algorithms and AI experiments easy.
.. figure:: https://user-images.githubusercontent.com/91237924/215658767-0e0fc221-aed0-4db1-9a82-6be55f244dba.png
:alt: ovms_high_level
@ -51,11 +70,10 @@ making deploying new algorithms and AI experiments easy.
To quickly start using OpenVINO™ Model Server, follow these steps:
Step 1: Prepare Docker
----------------------
Step 1: Prepare Docker `⇑ <#top>`__
###############################################################################################################################
Install `Docker Engine <https://docs.docker.com/engine/install/>`__,
including its
Install `Docker Engine <https://docs.docker.com/engine/install/>`__, including its
`post-installation <https://docs.docker.com/engine/install/linux-postinstall/>`__
steps, on your development system. To verify installation, test it,
using the following command. When it is ready, it will display a test
@ -92,11 +110,11 @@ image and a message.
Step 2: Preparing a Model Repository
------------------------------------
Step 2: Preparing a Model Repository `⇑ <#top>`__
###############################################################################################################################
The models need to be placed and mounted in a particular directory
structure and according to the following rules:
The models need to be placed and mounted in a particular directory structure and according to
the following rules:
::
@ -162,11 +180,13 @@ structure and according to the following rules:
model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.bin"
download_file(model_xml_url, XML_PATH, MODEL_DIR)
download_file(model_bin_url, BIN_PATH_name, MODEL_DIR)model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.xml"
model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.bin"
download_file(model_bin_url, BIN_PATH_name, MODEL_DIR)
download_file(model_xml_url, model_xml_name, base_model_dir)
download_file(model_bin_url, model_bin_name, base_model_dir)
model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.xml"
model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.bin"
download_file(model_xml_url, model_xml_name, base_model_dir)
download_file(model_bin_url, model_bin_name, base_model_dir)
.. parsed-literal::
@ -174,8 +194,8 @@ structure and according to the following rules:
Model Copied to "./models/detection/1".
Step 3: Start the Model Server Container
----------------------------------------
Step 3: Start the Model Server Container `⇑ <#top>`__
###############################################################################################################################
Pull and start the container:
@ -202,8 +222,8 @@ Check whether the OVMS container is running normally:
The required Model Server parameters are listed below. For additional
configuration options, see the `Model Server Parameters
section <https://docs.openvino.ai/2023.0/ovms_docs_parameters.html#doxid-ovms-docs-parameters>`__.
configuration options, see the
`Model Server Parameters section <https://docs.openvino.ai/2023.0/ovms_docs_parameters.html#doxid-ovms-docs-parameters>`__.
.. raw:: html
@ -419,7 +439,7 @@ openvino/model_server:latest
.. container:: line
represents the image name; the ovms binary is the Docker entry
represents the image name; the OVMS binary is the Docker entry
point
.. container:: line
@ -623,8 +643,8 @@ openvino/model_server:latest
If the serving port ``9000`` is already in use, please switch it to
another available port on your system. For example:\ ``-p 9020:9000``
Step 4: Prepare the Example Client Components
---------------------------------------------
Step 4: Prepare the Example Client Components `⇑ <#top>`__
###############################################################################################################################
OpenVINO Model Server exposes two sets of APIs: one compatible with
``TensorFlow Serving`` and another one, with ``KServe API``, for
@ -634,8 +654,9 @@ into existing systems the already leverage one of these APIs for
inference. This example will demonstrate how to write a TensorFlow
Serving API client for object detection.
Prerequisites
~~~~~~~~~~~~~
Prerequisites `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Install necessary packages.
@ -669,8 +690,9 @@ Install necessary packages.
You should consider upgrading via the '/home/adrian/repos/openvino_notebooks_adrian/venv/bin/python -m pip install --upgrade pip' command.
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -679,8 +701,9 @@ Imports
import matplotlib.pyplot as plt
from ovmsclient import make_grpc_client
Request Model Status
~~~~~~~~~~~~~~~~~~~~
Request Model Status `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -697,8 +720,9 @@ Request Model Status
{1: {'state': 'AVAILABLE', 'error_code': 0, 'error_message': 'OK'}}
Request Model Metadata
~~~~~~~~~~~~~~~~~~~~~~
Request Model Metadata `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -711,8 +735,9 @@ Request Model Metadata
{'model_version': 1, 'inputs': {'image': {'shape': [1, 3, 704, 704], 'dtype': 'DT_FLOAT'}}, 'outputs': {'1469_1470.0': {'shape': [-1], 'dtype': 'DT_FLOAT'}, '1078_1079.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1330_1331.0': {'shape': [36], 'dtype': 'DT_FLOAT'}, 'labels': {'shape': [-1], 'dtype': 'DT_INT32'}, '1267_1268.0': {'shape': [121], 'dtype': 'DT_FLOAT'}, '1141_1142.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1204_1205.0': {'shape': [484], 'dtype': 'DT_FLOAT'}, 'boxes': {'shape': [-1, 5], 'dtype': 'DT_FLOAT'}}}
Load input image
~~~~~~~~~~~~~~~~
Load input image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -742,8 +767,9 @@ Load input image
.. image:: 117-model-server-with-output_files/117-model-server-with-output_20_1.png
Request Prediction on a Numpy Array
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Request Prediction on a Numpy Array `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -767,8 +793,9 @@ Request Prediction on a Numpy Array
[2.2261986e+01 4.5406548e+01 1.8868817e+02 1.0225631e+02 3.0407205e-01]]
Visualization
~~~~~~~~~~~~~
Visualization `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -850,9 +877,11 @@ command:
ovms
References
----------
References `⇑ <#top>`__
###############################################################################################################################
1. `OpenVINO™ Model
Server <https://docs.openvino.ai/2023.0/ovms_what_is_openvino_model_server.html>`__
2. `openvinotoolkit/model_server <https://github.com/openvinotoolkit/model_server/>`__
1. `OpenVINO™ Model Server
documentation <https://docs.openvino.ai/2023.0/ovms_what_is_openvino_model_server.html>`__
2. `OpenVINO™ Model Server GitHub
repository <https://github.com/openvinotoolkit/model_server/>`__

8
docs/notebooks/117-model-server-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/117-model-server-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/117-model-server-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/117-model-server-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="117-model-server-with-output_20_1.png">117-model-server-with-output_20_1.png</a> 12-Jul-2023 00:11 112408
<a href="117-model-server-with-output_25_1.png">117-model-server-with-output_25_1.png</a> 12-Jul-2023 00:11 232667
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/117-model-server-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="117-model-server-with-output_20_1.png">117-model-server-with-output_20_1.png</a> 16-Aug-2023 01:31 112408
<a href="117-model-server-with-output_25_1.png">117-model-server-with-output_25_1.png</a> 16-Aug-2023 01:31 232667
</pre><hr></body>
</html>

256
docs/notebooks/118-optimize-preprocessing-with-output.rst
View File

@ -1,6 +1,8 @@
Optimize Preprocessing
======================
.. _top:
When input data does not fit the model input tensor perfectly,
additional operations/steps are needed to transform the data to the
format expected by the model. This tutorial demonstrates how it could be
@ -15,18 +17,59 @@ and
This tutorial include following steps:
- Downloading the model.
- Setup preprocessing with ModelOptimizer, loading the model and inference with original image.
- Setup preprocessing with Preprocessing API, loading the model and inference with original image.
- Fitting image to the model input type and inference with prepared image.
- Comparing results on one picture.
- Comparing performance.
- Downloading the model.
- Setup preprocessing with model conversion API, loading the model and
inference with original image.
- Setup preprocessing with Preprocessing API, loading the model and
inference with original image.
- Fitting image to the model input type and inference with prepared
image.
- Comparing results on one picture.
- Comparing performance.
Settings
--------
**Table of contents**:
- `Settings <#settings>`__
- `Imports <#imports>`__
- `Setup image and device <#setup-image-and-device>`__
- `Downloading the model <#downloading-the-model>`__
- `Create core <#create-core>`__
- `Check the original parameters of image <#check-the-original-parameters-of-image>`__
- `Convert model to OpenVINO IR and setup preprocessing steps with model conversion API <#convert-model-to-openvino-ir-and-setup-preprocessing-steps-with-model-conversion-api>`__
- `Prepare image <#prepare-image>`__
- `Compile model and perform inference <#compile-model-and-perform-inference>`__
- `Setup preprocessing steps with Preprocessing API and perform inference <#setup-preprocessing-steps-with-preprocessing-api-and-perform-inference>`__
- `Convert model to OpenVINO IR with model conversion API <#convert-model-to-openvino-ir-with-model-conversion-api>`__
- `Create PrePostProcessor Object <#create-prepostprocessor-object>`__
- `Declare User’s Data Format <#declare-users-data-format>`__
- `Declaring Model Layout <#declaring-model-layout>`__
- `Preprocessing Steps <#preprocessing-steps>`__
- `Integrating Steps into a Model <#integrating-steps-into-a-model>`__
- `Load model and perform inference <#load-model-and-perform-inference>`__
- `Fit image manually and perform inference <#fit-image-manually-and-perform-inference>`__
- `Load the model <#load-the-model>`__
- `Load image and fit it to model input <#load-image-and-fit-it-to-model-input>`__
- `Perform inference <#perform-inference>`__
- `Compare results <#compare-results>`__
- `Compare results on one image <#compare-results-on-one-image>`__
- `Compare performance <#compare-performance>`__
Settings `⇑ <#top>`__
###############################################################################################################################
Imports `⇑ <#top>`__
###############################################################################################################################
Imports
-------
.. code:: ipython3
@ -43,14 +86,15 @@ Imports
.. parsed-literal::
2023-07-11 22:53:33.947684: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-11 22:53:33.981920: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2023-08-15 22:57:19.952994: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-15 22:57:19.987688: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-11 22:53:34.528705: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-15 22:57:20.520711: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Setup image and device
~~~~~~~~~~~~~~~~~~~~~~
Setup image and device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -79,8 +123,9 @@ Setup image and device
Downloading the model
~~~~~~~~~~~~~~~~~~~~~
Downloading the model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
This tutorial uses the
`InceptionResNetV2 <https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_resnet_v2>`__.
@ -111,7 +156,7 @@ and save it to the disk.
.. parsed-literal::
2023-07-11 22:53:35.902963: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
2023-08-15 22:57:21.888060: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
@ -135,15 +180,17 @@ and save it to the disk.
INFO:tensorflow:Assets written to: model/InceptionResNetV2/assets
Create core
~~~~~~~~~~~
Create core `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
core = Core()
Check the original parameters of image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Check the original parameters of image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -160,16 +207,15 @@ Check the original parameters of image
.. image:: 118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_12_1.png
.. image:: 118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_13_1.png
Convert model to OpenVINO IR and setup preprocessing steps with Model Optimizer
-------------------------------------------------------------------------------
Convert model to OpenVINO IR and setup preprocessing steps with model conversion API. `⇑ <#top>`__
###############################################################################################################################
Use Model Optimizer to convert a TensorFlow model to OpenVINO IR.
``mo.convert_model`` python function will be used for converting model
using `OpenVINO Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__.
To convert a TensorFlow model to OpenVINO IR, use the
``mo.convert_model`` python function of `model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
The function returns instance of OpenVINO Model class, which is ready to
use in Python interface but can also be serialized to OpenVINO IR format
for future execution using ``openvino.runtime.serialize``. The models
@ -182,9 +228,12 @@ pre-processing sub-graphs into the converted model.
Setup the following conversions:
- mean normalization with ``mean_values`` parameter
- scale with ``scale_values``
- color conversion, the color format of example image will be ``BGR``, but the model required ``RGB`` format, so add ``reverse_input_channels=True`` to process the image into the desired format
- mean normalization with ``mean_values`` parameter.
- scale with ``scale_values``.
- color conversion, the color format of example image will be ``BGR``,
but the model required ``RGB`` format, so add
``reverse_input_channels=True`` to process the image into the desired
format.
Also converting of layout could be specified with ``layout`` option.
More information and parameters described in the `Embedding
@ -209,8 +258,9 @@ article <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Additional_Optimiza
input_shape=[1,299,299,3])
serialize(ov_model_mo_preprocess, str(ir_path_mo_preprocess))
Prepare image
~~~~~~~~~~~~~
Prepare image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -245,8 +295,9 @@ Prepare image
The data type of the image is float32
Compile model and perform inerence
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compile model and perform inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -256,25 +307,35 @@ Compile model and perform inerence
result = compiled_model_mo_pp(mo_pp_input_tensor)[output_layer]
Setup preprocessing steps with Preprocessing API and perform inference
----------------------------------------------------------------------
Setup preprocessing steps with Preprocessing API and perform inference. `⇑ <#top>`__
###############################################################################################################################
Intuitively, preprocessing API consists of the following parts:
- Tensor - declares user data format, like shape, layout, precision, color format from actual user’s data.
- Steps - describes sequence of preprocessing steps which need to be applied to user data.
- Model - specifies model data format. Usually, precision and shape are already known for model, only additional information, like layout can be specified.
- Tensor - declares user data format, like shape, layout, precision,
color format from actual user’s data.
- Steps - describes sequence of preprocessing steps which need to be
applied to user data.
- Model - specifies model data format. Usually, precision and shape are
already known for model, only additional information, like layout can
be specified.
Graph modifications of a model shall be performed after the model is
read from a drive and before it is loaded on the actual device.
Pre-processing support following operations (please, see more details
`here <https://docs.openvino.ai/2023.0/classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1aeacaf406d72a238e31a359798ebdb3b7>`__)
- Mean/Scale Normalization - Converting Precision - Converting layout
(transposing) - Resizing Image - Color Conversion - Custom Operations
Convert model to OpenVINO IR with Model Optimizer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Mean/Scale Normalization
- Converting Precision
- Converting layout (transposing)
- Resizing Image
- Color Conversion
- Custom Operations
Convert model to OpenVINO IR with model conversion API `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The options for preprocessing are not required.
@ -292,8 +353,9 @@ The options for preprocessing are not required.
input_shape=[1,299,299,3])
serialize(ppp_model, str(ir_path))
Create PrePostProcessor Object
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create ``PrePostProcessor`` Object `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The
`PrePostProcessor() <https://docs.openvino.ai/2023.0/classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>`__
@ -306,8 +368,9 @@ a model.
ppp = PrePostProcessor(ppp_model)
Declare User’s Data Format
~~~~~~~~~~~~~~~~~~~~~~~~~~
Declare User’s Data Format `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To address particular input of a model/preprocessor, use the
``PrePostProcessor.input(input_name)`` method. If the model has only one
@ -324,9 +387,11 @@ for more information about parameters for overriding.
Below is all the specified input information:
- Precision is ``U8``(unsigned 8-bit integer).
- Size is non-fixed, setup of one determined shape size can be done with ``.set_shape([1, 577, 800, 3])``.
- Layout is ``“NHWC”``. It means, for example: height=577, width=800, channels=3.
- Precision is ``U8`` (unsigned 8-bit integer).
- Size is non-fixed, setup of one determined shape size can be done
with ``.set_shape([1, 577, 800, 3])``
- Layout is ``“NHWC”``. It means, for example: height=577, width=800,
channels=3.
The height and width are necessary for resizing, and channels are needed
for mean/scale normalization.
@ -345,12 +410,13 @@ for mean/scale normalization.
.. parsed-literal::
<openvino._pyopenvino.preprocess.InputTensorInfo at 0x7efb4037f5b0>
<openvino._pyopenvino.preprocess.InputTensorInfo at 0x7fbb8c094ab0>
Declaring Model Layout
~~~~~~~~~~~~~~~~~~~~~~
Declaring Model Layout `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Model input already has information about precision and shape.
Preprocessing API is not intended to modify this. The only thing that
@ -374,12 +440,13 @@ may be specified is input data
.. parsed-literal::
<openvino._pyopenvino.preprocess.InputModelInfo at 0x7efc3610ac70>
<openvino._pyopenvino.preprocess.InputModelInfo at 0x7fbae64da530>
Preprocessing Steps
~~~~~~~~~~~~~~~~~~~
Preprocessing Steps `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Now, the sequence of preprocessing steps can be defined. For more
information about preprocessing steps, see
@ -387,10 +454,14 @@ information about preprocessing steps, see
Perform the following:
- Convert ``U8`` to ``FP32`` precision.
- Resize to height/width of a model. Be aware that if a model accepts dynamic size, for example, ``{?, 3, ?, ?}`` resize will not know how to resize the picture. Therefore, in this case, target height/ width should be specified. For more details, see also the `PreProcessSteps.resize() <https://docs.openvino.ai/2023.0/classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1a40dab78be1222fee505ed6a13400efe6>`__.
- Subtract mean from each channel.
- Divide each pixel data to appropriate scale value.
- Convert ``U8`` to ``FP32`` precision.
- Resize to height/width of a model. Be aware that if a model accepts
dynamic size, for example, ``{?, 3, ?, ?}`` resize will not know how
to resize the picture. Therefore, in this case, target height/ width
should be specified. For more details, see also the
`PreProcessSteps.resize() <https://docs.openvino.ai/2023.0/classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1a40dab78be1222fee505ed6a13400efe6>`__.
- Subtract mean from each channel.
- Divide each pixel data to appropriate scale value.
There is no need to specify conversion layout. If layouts are different,
then such conversion will be added explicitly.
@ -409,16 +480,17 @@ then such conversion will be added explicitly.
.. parsed-literal::
<openvino._pyopenvino.preprocess.PreProcessSteps at 0x7efd2c4155b0>
<openvino._pyopenvino.preprocess.PreProcessSteps at 0x7fbb2c6c16f0>
Integrating Steps into a Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Integrating Steps into a Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Once the preprocessing steps have been finished, the model can be
finally built. It is possible to display PrePostProcessor configuration
for debugging purposes.
finally built. It is possible to display ``PrePostProcessor``
configuration for debugging purposes.
.. code:: ipython3
@ -439,8 +511,9 @@ for debugging purposes.
Load model and perform inference
--------------------------------
Load model and perform inference `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -457,19 +530,22 @@ Load model and perform inference
ppp_input_tensor = prepare_image_api_preprocess(image_path)
results = compiled_model_with_preprocess_api(ppp_input_tensor)[ppp_output_layer][0]
Fit image manually and perform inference
----------------------------------------
Fit image manually and perform inference `⇑ <#top>`__
###############################################################################################################################
Load the model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load the model
~~~~~~~~~~~~~~
.. code:: ipython3
model = core.read_model(model=ir_path)
compiled_model = core.compile_model(model=model, device_name=device.value)
Load image and fit it to model input
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load image and fit it to model input `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -502,8 +578,9 @@ Load image and fit it to model input
The data type of the image is float32
Perform inference
~~~~~~~~~~~~~~~~~
Perform inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -511,11 +588,13 @@ Perform inference
result = compiled_model(input_tensor)[output_layer]
Compare results
---------------
Compare results `⇑ <#top>`__
###############################################################################################################################
Compare results on one image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Compare results on one image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -538,7 +617,7 @@ Compare results on one image
imagenet_classes = ['background'] + imagenet_classes
# get result for inference with preprocessing api
print("Result of inference for preprocessing with ModelOptimizer:")
print("Result of inference for preprocessing with Model Optimizer:")
res = check_results(mo_pp_input_tensor, compiled_model_mo_pp, imagenet_classes)
print("\n")
@ -556,7 +635,7 @@ Compare results on one image
.. parsed-literal::
Result of inference for preprocessing with ModelOptimizer:
Result of inference for preprocessing with Model Optimizer:
n02099601 golden retriever, 0.56439
n02098413 Lhasa, Lhasa apso, 0.35731
n02108915 French bulldog, 0.00730
@ -580,8 +659,9 @@ Compare results on one image
n02100877 Irish setter, red setter, 0.00116
Compare performance
~~~~~~~~~~~~~~~~~~~
Compare performance `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -621,7 +701,7 @@ Compare performance
.. parsed-literal::
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0200 seconds per image, FPS: 49.90
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0153 seconds per image, FPS: 65.52
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0187 seconds per image, FPS: 53.59
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0199 seconds per image, FPS: 50.13
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0155 seconds per image, FPS: 64.58
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0188 seconds per image, FPS: 53.27

0
docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_12_1.png → docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_13_1.png
View File

6
docs/notebooks/118-optimize-preprocessing-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/118-optimize-preprocessing-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/118-optimize-preprocessing-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/118-optimize-preprocessing-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="118-optimize-preprocessing-with-output_12_1.png">118-optimize-preprocessing-with-output_12_1.png</a> 12-Jul-2023 00:11 387941
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/118-optimize-preprocessing-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="118-optimize-preprocessing-with-output_13_1.png">118-optimize-preprocessing-with-output_13_1.png</a> 16-Aug-2023 01:31 387941
</pre><hr></body>
</html>

122
docs/notebooks/119-tflite-to-openvino-with-output.rst
View File

@ -1,25 +1,45 @@
Convert a Tensorflow Lite Model to OpenVINO™
============================================
.. _top:
`TensorFlow Lite <https://www.tensorflow.org/lite/guide>`__, often
referred to as TFLite, is an open source library developed for deploying
machine learning models to edge devices.
This short tutorial shows how to convert a TensorFlow Lite
`efficientnet-lite-b0 <https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/fp32/2>`__
`EfficientNet-Lite-B0 <https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/fp32/2>`__
image classification model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
Runtime <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
Preparation
-----------
**Table of contents**:
- `Preparation <#preparation>`__
- `Install requirements <#install-requirements>`__
- `Imports <#imports>`__
- `Download TFLite model <#download-tflite-model>`__
- `Convert a Model to OpenVINO IR Format <#convert-a-model-to-openvino-ir-format>`__
- `Load model using OpenVINO TensorFlow Lite Frontend <#load-model-using-openvino-tensorflow-lite-frontend>`__
- `Run OpenVINO model inference <#run-openvino-model-inference>`__
- `Select inference device <#select-inference-device>`__
- `Estimate Model Performance <#estimate-model-performance>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Install requirements
~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -33,8 +53,9 @@ Install requirements
filename='notebook_utils.py'
);
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -46,8 +67,9 @@ Imports
from notebook_utils import download_file, load_image
Download TFLite model
---------------------
Download TFLite model `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -70,25 +92,26 @@ Download TFLite model
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite')
Convert a Model to OpenVINO IR Format
-------------------------------------
Convert a Model to OpenVINO IR Format `⇑ <#top>`__
###############################################################################################################################
To convert the TFLite model to OpenVINO IR, OpenVINO Model Optimizer
Python API can be used. ``mo.convert_model`` function accept path to
TFLite model and returns OpenVINO Model class instance which represents
this model. Obtained model is ready to use and loading on device using
``compile_model`` or can be saved on disk using ``serialize`` function
reducing loading time for next running. Optionally, we can apply
compression to FP16 model weigths using ``compress_to_fp16=True`` option
and integrate preprocessing using this approach. See the `Model
Optimizer Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__
for more information about Model Optimizer and TensorFlow Lite `models
suport <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite.html>`__.
To convert the TFLite model to OpenVINO IR, model conversion Python API
can be used. ``mo.convert_model`` function accepts the path to the
TFLite model and returns an OpenVINO Model class instance which
represents this model. The obtained model is ready to use and to be
loaded on a device using ``compile_model`` or can be saved on a disk
using ``serialize`` function, reducing loading time for next running.
Optionally, we can apply compression to the FP16 model weights, using
the ``compress_to_fp16=True`` option and integrate preprocessing using
this approach. For more information about model conversion, see this
`page <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__.
For TensorFlow Lite models support, refer to this
`tutorial <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite.html>`__.
.. code:: ipython3
@ -102,10 +125,11 @@ suport <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_conver
Model model/efficientnet_lite0_fp32_2.tflite successfully converted and saved to model/efficientnet_lite0_fp32_2.xml
Load model using OpenVINO TensorFlow Lite Frontend
--------------------------------------------------
Load model using OpenVINO TensorFlow Lite Frontend `⇑ <#top>`__
###############################################################################################################################
TensorFlow Lite models are supported via FrontEnd API. You may skip
TensorFlow Lite models are supported via ``FrontEnd`` API. You may skip
conversion to IR and read models directly by OpenVINO runtime API. For
more examples supported formats reading via Frontend API, please look
this `tutorial <../002-openvino-api>`__.
@ -116,8 +140,9 @@ this `tutorial <../002-openvino-api>`__.
ov_model = core.read_model(tflite_model_path)
Run OpenVINO model inference
----------------------------
Run OpenVINO model inference `⇑ <#top>`__
###############################################################################################################################
We can find information about model input preprocessing in its
`description <https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/fp32/2>`__
@ -131,10 +156,11 @@ on `TensorFlow Hub <https://tfhub.dev/>`__.
resized_image = image.resize((224, 224))
input_tensor = np.expand_dims((np.array(resized_image).astype(np.float32) - 127) / 128, 0)
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -191,14 +217,14 @@ select device from dropdown list for running inference using OpenVINO
Predicted label: n02109047 Great Dane with probability 0.715318
Estimate Model Performance
--------------------------
Estimate Model Performance `⇑ <#top>`__
###############################################################################################################################
`Benchmark
Tool <https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html>`__
`Benchmark Tool <https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html>`__
is used to measure the inference performance of the model on CPU and
GPU.
**NOTE**: For more accurate performance, it is recommended to run
``benchmark_app`` in a terminal/command prompt after closing other
applications. Run ``benchmark_app -m model.xml -d CPU`` to benchmark
@ -233,7 +259,7 @@ GPU.
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 21.86 ms
[ INFO ] Read model took 9.14 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,224,224,3]
@ -247,7 +273,7 @@ GPU.
[ INFO ] Model outputs:
[ INFO ] Softmax (node: 61) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 183.81 ms
[ INFO ] Compile model took 151.57 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: TensorFlow_Lite_Frontend_IR
@ -269,15 +295,15 @@ GPU.
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.41 ms
[ INFO ] First inference took 7.60 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 17364 iterations
[ INFO ] Duration: 15009.20 ms
[ INFO ] Count: 17526 iterations
[ INFO ] Duration: 15005.75 ms
[ INFO ] Latency:
[ INFO ] Median: 5.04 ms
[ INFO ] Average: 5.05 ms
[ INFO ] Min: 3.22 ms
[ INFO ] Max: 15.36 ms
[ INFO ] Throughput: 1156.89 FPS
[ INFO ] Median: 5.00 ms
[ INFO ] Average: 5.00 ms
[ INFO ] Min: 3.28 ms
[ INFO ] Max: 14.83 ms
[ INFO ] Throughput: 1167.95 FPS

8
docs/notebooks/119-tflite-to-openvino-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/119-tflite-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/119-tflite-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/119-tflite-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="119-tflite-to-openvino-with-output_16_1.jpg">119-tflite-to-openvino-with-output_16_1.jpg</a> 12-Jul-2023 00:11 68170
<a href="119-tflite-to-openvino-with-output_16_1.png">119-tflite-to-openvino-with-output_16_1.png</a> 12-Jul-2023 00:11 621006
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/119-tflite-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="119-tflite-to-openvino-with-output_16_1.jpg">119-tflite-to-openvino-with-output_16_1.jpg</a> 16-Aug-2023 01:31 68170
<a href="119-tflite-to-openvino-with-output_16_1.png">119-tflite-to-openvino-with-output_16_1.png</a> 16-Aug-2023 01:31 621006
</pre><hr></body>
</html>

150
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst
View File

@ -1,6 +1,8 @@
Convert a TensorFlow Object Detection Model to OpenVINO™
========================================================
.. _top:
`TensorFlow <https://www.tensorflow.org/>`__, or TF for short, is an
open-source framework for machine learning.
@ -21,11 +23,33 @@ Representation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_IR_and_opset
(OpenVINO IR) format, using `Model
Optimizer <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
Runtime <https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
**Table of contents**:
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Download Model from TensorFlow Hub <#download-model-from-tensorflow-hub>`__
- `Convert Model to OpenVINO IR <#convert-model-to-openvino-ir>`__
- `Test Inference on the Converted Model <#test-inference-on-the-converted-model>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Get Model Information <#get-model-information>`__
- `Get an Image for Test Inference <#get-an-image-for-test-inference>`__
- `Perform Inference <#perform-inference>`__
- `Inference Result Visualization <#inference-result-visualization>`__
- `Next Steps <#next-steps>`__
- `Async inference pipeline <#async-inference-pipeline>`__
- `Integration preprocessing to model <#integration-preprocessing-to-model>`__
Prerequisites `⇑ <#top>`__
###############################################################################################################################
Prerequisites
-------------
Install required packages:
@ -46,8 +70,9 @@ The notebook uses utility functions. The cell below will download the
filename="notebook_utils.py",
);
Imports
-------
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -66,8 +91,9 @@ Imports
from openvino.runtime import Core, serialize
from openvino.tools import mo
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
Define model related variables and create corresponding directories:
@ -93,8 +119,9 @@ Define model related variables and create corresponding directories:
tf_model_archive_filename = f"{model_name}.tar.gz"
Download Model from TensorFlow Hub
----------------------------------
Download Model from TensorFlow Hub `⇑ <#top>`__
###############################################################################################################################
Download archive with TensorFlow Object Detection model
(`faster_rcnn_resnet50_v1_640x640 <https://tfhub.dev/tensorflow/faster_rcnn/resnet50_v1_640x640/1>`__)
@ -119,7 +146,7 @@ from TensorFlow Hub:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz')
@ -132,8 +159,9 @@ Extract TensorFlow Object Detection model from the downloaded archive:
with tarfile.open(tf_model_dir / tf_model_archive_filename) as file:
file.extractall(path=tf_model_dir)
Convert Model to OpenVINO IR
----------------------------
Convert Model to OpenVINO IR `⇑ <#top>`__
###############################################################################################################################
OpenVINO Model Optimizer Python API can be used to convert the
TensorFlow model to OpenVINO IR.
@ -143,7 +171,7 @@ returns OpenVINO Model class instance which represents this model. Also
we need to provide model input shape (``input_shape``) that is described
at `model overview page on TensorFlow
Hub <https://tfhub.dev/tensorflow/faster_rcnn/resnet50_v1_640x640/1>`__.
Optionally, we can apply compression to FP16 model weigths using
Optionally, we can apply compression to FP16 model weights using
``compress_to_fp16=True`` option and integrate preprocessing using this
approach.
@ -154,7 +182,7 @@ when the model is run in the future.
See the `Model Optimizer Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__
for more information about Model Optimizer and TensorFlow `models
suport <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
support <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
.. code:: ipython3
@ -166,13 +194,15 @@ suport <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_conver
# Save converted OpenVINO IR model to the corresponding directory
serialize(ov_model, openvino_ir_path)
Test Inference on the Converted Model
-------------------------------------
Test Inference on the Converted Model `⇑ <#top>`__
###############################################################################################################################
Select inference device
-----------------------
select device from dropdown list for running inference using OpenVINO
Select inference device `⇑ <#top>`__
###############################################################################################################################
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -197,8 +227,9 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
~~~~~~~~~~~~~~
Load the Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -206,8 +237,9 @@ Load the Model
openvino_ir_model = core.read_model(openvino_ir_path)
compiled_model = core.compile_model(model=openvino_ir_model, device_name=device.value)
Get Model Information
~~~~~~~~~~~~~~~~~~~~~
Get Model Information `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Faster R-CNN with Resnet-50 V1 object detection model has one input - a
three-channel image of variable size. The input tensor shape is
@ -215,14 +247,22 @@ three-channel image of variable size. The input tensor shape is
Model output dictionary contains several tensors:
- ``num_detections`` - the number of detections in ``[N]`` format.
- ``detection_boxes`` - bounding box coordinates for all ``N`` detections in ``[ymin, xmin, ymax, xmax]`` format.
- ``detection_classes`` - ``N`` detection class indexes size from the label file.
- ``detection_scores``- ``N`` detection scores (confidence) for each detected class.
- ``raw_detection_boxes`` - decoded detection boxes without Non-Max suppression.
- ``raw_detection_scores`` - class score logits for raw detection boxes.
- ``detection_anchor_indices`` - the anchor indices of the detections after NMS.
- ``detection_multiclass_scores`` - class score distribution (including background) for detection boxes in the image including background class.
- ``num_detections`` - the number of detections in ``[N]`` format.
- ``detection_boxes`` - bounding box coordinates for all ``N``
detections in ``[ymin, xmin, ymax, xmax]`` format.
- ``detection_classes`` - ``N`` detection class indexes size from the
label file.
- ``detection_scores`` - ``N`` detection scores (confidence) for each
detected class.
- ``raw_detection_boxes`` - decoded detection boxes without Non-Max
suppression.
- ``raw_detection_scores`` - class score logits for raw detection
boxes.
- ``detection_anchor_indices`` - the anchor indices of the detections
after NMS.
- ``detection_multiclass_scores`` - class score distribution (including
background) for detection boxes in the image including background
class.
In this tutorial we will mostly use ``detection_boxes``,
``detection_classes``, ``detection_scores`` tensors. It is important to
@ -266,8 +306,9 @@ for more information about model inputs, outputs and their formats.
<ConstOutput: names[raw_detection_scores] shape[1,300,91] type: f32>
Get an Image for Test Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Get an Image for Test Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load and save an image:
@ -292,7 +333,7 @@ Load and save an image:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg')
@ -320,7 +361,7 @@ Read the image, resize and convert it to the input shape of the network:
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f877c319b50>
<matplotlib.image.AxesImage at 0x7f9b48184ca0>
@ -328,8 +369,9 @@ Read the image, resize and convert it to the input shape of the network:
.. image:: 120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_25_1.png
Perform Inference
~~~~~~~~~~~~~~~~~
Perform Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -439,8 +481,9 @@ outputs will be used.
image_detections_num: [300.]
Inference Result Visualization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inference Result Visualization `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Define utility functions to visualize the inference results
@ -580,7 +623,7 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt')
@ -619,24 +662,25 @@ original test image:
.. image:: 120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png
Next Steps
----------
Next Steps `⇑ <#top>`__
###############################################################################################################################
This section contains suggestions on how to additionally improve the
performance of your application using OpenVINO.
Async inference pipeline
~~~~~~~~~~~~~~~~~~~~~~~~
Async inference pipeline `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The key advantage of the Async API is that when a device is busy with
inference, the application can perform other tasks in parallel (for
example, populating inputs or scheduling other requests) rather than
wait for the current inference to complete first. To understand how to
perform async inference using openvino, refer to the `Async API
tutorial <115-async-api-with-output.html>`__.
The key advantage of the Async API is that when a device is busy with inference,
the application can perform other tasks in parallel (for example, populating inputs or
scheduling other requests) rather than wait for the current inference to
complete first. To understand how to perform async inference using
openvino, refer to the `Async API tutorial <115-async-api-with-output.html>`__.
Integration preprocessing to model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Integration preprocessing to model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Preprocessing API enables making preprocessing a part of the model
reducing application code and dependency on additional image processing

4
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59da4a179a69c8527e526a4b705f9d2ac5225cb90e2489353e36340dd12b481c
size 391541
oid sha256:b7874b5d950db3016c015449880bbc25d55dcbb66b14f1fd54593352f044b3a2
size 391330

8
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-object-detection-to-openvino-with-output_25_1.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 12-Jul-2023 00:11 395346
<a href="120-tensorflow-object-detection-to-openvino-with-output_38_0.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 12-Jul-2023 00:11 391541
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-object-detection-to-openvino-with-output_25_1.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 16-Aug-2023 01:31 395346
<a href="120-tensorflow-object-detection-to-openvino-with-output_38_0.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 16-Aug-2023 01:31 391330
</pre><hr></body>
</html>

1474
docs/notebooks/121-convert-to-openvino-with-output.rst Normal file
View File

File diff suppressed because it is too large Load Diff

116
docs/notebooks/201-vision-monodepth-with-output.rst
View File

@ -1,6 +1,8 @@
Monodepth Estimation with OpenVINO
==================================
.. _top:
This tutorial demonstrates Monocular Depth Estimation with MidasNet in
OpenVINO. Model information can be found
`here <https://docs.openvino.ai/2023.0/omz_models_model_midasnet.html>`__.
@ -26,13 +28,39 @@ Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot
Cross-dataset
Transfer,” <https://ieeexplore.ieee.org/document/9178977>`__ in IEEE
Transactions on Pattern Analysis and Machine Intelligence, doi:
10.1109/TPAMI.2020.3019967.
``10.1109/TPAMI.2020.3019967``.
Preparation
-----------
**Table of contents**:
- `Preparation <#preparation>`__
- `Install requirements <#install-requirements>`__
- `Imports <#imports>`__
- `Download the model <#download-the-model>`__
- `Functions <#functions>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Monodepth on Image <#monodepth-on-image>`__
- `Load, resize and reshape input image <#load-resize-and-reshape-input-image>`__
- `Do inference on the image <#do-inference-on-the-image>`__
- `Display monodepth image <#display-monodepth-image>`__
- `Monodepth on Video <#monodepth-on-video>`__
- `Video Settings <#video-settings>`__
- `Load the Video <#load-the-video>`__
- `Do Inference on a Video and Create Monodepth Video <#do-inference-on-a-video-and-create-monodepth-video>`__
- `Display Monodepth Video <#display-monodepth-video>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Install requirements
~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -51,12 +79,13 @@ Install requirements
.. parsed-literal::
('notebook_utils.py', <http.client.HTTPMessage at 0x7f341c4d7040>)
('notebook_utils.py', <http.client.HTTPMessage at 0x7ffb1c728ca0>)
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -80,8 +109,9 @@ Imports
from notebook_utils import download_file, load_image
Download the model
~~~~~~~~~~~~~~~~~~
Download the model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -109,8 +139,9 @@ Download the model
model/MiDaS_small.bin: 0%| | 0.00/31.6M [00:00<?, ?B/s]
Functions
---------
Functions `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -142,10 +173,11 @@ Functions
"""
return cv2.cvtColor(image_data, cv2.COLOR_BGR2RGB)
Select inference device
-----------------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -170,8 +202,9 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------
Load the Model `⇑ <#top>`__
###############################################################################################################################
Load the model in OpenVINO Runtime with ``ie.read_model`` and compile it
for the specified device with ``ie.compile_model``. Get input and output
@ -190,11 +223,13 @@ keys and the expected input shape for the model.
network_input_shape = list(input_key.shape)
network_image_height, network_image_width = network_input_shape[2:]
Monodepth on Image
------------------
Monodepth on Image `⇑ <#top>`__
###############################################################################################################################
Load, resize and reshape input image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load, resize and reshape input image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The input image is read with OpenCV, resized to network input size, and
reshaped to (N,C,H,W) (N=number of images, C=number of channels,
@ -211,8 +246,9 @@ H=height, W=width).
# Reshape the image to network input shape NCHW.
input_image = np.expand_dims(np.transpose(resized_image, (2, 0, 1)), 0)
Do inference on the image
~~~~~~~~~~~~~~~~~~~~~~~~~
Do inference on the image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Do inference, convert the result to an image, and resize it to the
original image shape.
@ -229,8 +265,9 @@ original image shape.
# in (width, height), [::-1] reverses the (height, width) shape to match this.
result_image = cv2.resize(result_image, image.shape[:2][::-1])
Display monodepth image
~~~~~~~~~~~~~~~~~~~~~~~
Display monodepth image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -243,15 +280,17 @@ Display monodepth image
.. image:: 201-vision-monodepth-with-output_files/201-vision-monodepth-with-output_18_0.png
Monodepth on Video
------------------
Monodepth on Video `⇑ <#top>`__
###############################################################################################################################
By default, only the first 100 frames are processed in order to quickly
check that everything works. Change ``NUM_FRAMES`` in the cell below to
modify this. Set ``NUM_FRAMES`` to 0 to process the whole video.
Video Settings
~~~~~~~~~~~~~~
Video Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -279,8 +318,9 @@ Video Settings
output_directory.mkdir(exist_ok=True)
result_video_path = output_directory / f"{Path(VIDEO_FILE).stem}_monodepth.mp4"
Load the Video
~~~~~~~~~~~~~~
Load the Video `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load the video from a ``VIDEO_FILE``, set in the *Video Settings* cell
above. Open the video to read the frame width and height and fps, and
@ -317,8 +357,9 @@ compute values for these properties for the monodepth video.
The monodepth video will be scaled with a factor 0.5, have width 320, height 180, and run at 15.00 fps
Do Inference on a Video and Create Monodepth Video
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Do Inference on a Video and Create Monodepth Video `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -417,12 +458,13 @@ Do Inference on a Video and Create Monodepth Video
.. parsed-literal::
Processed 60 frames in 35.09 seconds. Total FPS (including video processing): 1.71.Inference FPS: 42.97
Processed 60 frames in 48.08 seconds. Total FPS (including video processing): 1.25.Inference FPS: 42.27
Monodepth Video saved to 'output/Coco%20Walking%20in%20Berkeley_monodepth.mp4'.
Display Monodepth Video
~~~~~~~~~~~~~~~~~~~~~~~
Display Monodepth Video `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -445,7 +487,7 @@ Display Monodepth Video
.. parsed-literal::
Showing monodepth video saved at
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/201-vision-monodepth/output/Coco%20Walking%20in%20Berkeley_monodepth.mp4
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/201-vision-monodepth/output/Coco%20Walking%20in%20Berkeley_monodepth.mp4
If you cannot see the video in your browser, please click on the following link to download the video

6
docs/notebooks/201-vision-monodepth-with-output_files/index.html
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/201-vision-monodepth-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/201-vision-monodepth-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/201-vision-monodepth-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="201-vision-monodepth-with-output_18_0.png">201-vision-monodepth-with-output_18_0.png</a> 12-Jul-2023 00:11 959858
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/201-vision-monodepth-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="201-vision-monodepth-with-output_18_0.png">201-vision-monodepth-with-output_18_0.png</a> 16-Aug-2023 01:31 959858
</pre><hr></body>
</html>

143
docs/notebooks/202-vision-superresolution-image-with-output.rst
View File

@ -1,6 +1,8 @@
Single Image Super Resolution with OpenVINO™
============================================
.. _top:
Super Resolution is the process of enhancing the quality of an image by
increasing the pixel count using deep learning. This notebook shows the
Single Image Super Resolution (SISR) which takes just one low resolution
@ -12,22 +14,56 @@ based on the research paper cited below.
Y. Liu et al., `“An Attention-Based Approach for Single Image Super
Resolution,” <https://arxiv.org/abs/1807.06779>`__ 2018 24th
International Conference on Pattern Recognition (ICPR), 2018,
pp. 2777-2784, doi: 10.1109/ICPR.2018.8545760.
pp. 2777-2784, doi: 10.1109/ICPR.2018.8545760.
Preparation
-----------
**Table of contents**:
- `Preparation <#preparation>`__
- `Install requirements <#install-requirements>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Select inference device <#select-inference-device>`__
- `Functions <#functions>`__
- `Load the Superresolution Model <#load-the-superresolution-model>`__
- `Load and Show the Input Image <#load-and-show-the-input-image>`__
- `Superresolution on a Crop of the Image <#superresolution-on-a-crop-of-the-image>`__
- `Crop the Input Image once. <#crop-the-input-image-once>`__
- `Reshape/Resize Crop for Model Input <#reshape-resize-crop-for-model-input>`__
- `Do Inference <#do-inference>`__
- `Show and Save Results <#show-and-save-results>`__
- `Save Superresolution and Bicubic Image Crop <#save-superresolution-and-bicubic-image-crop>`__
- `Write Animated GIF with Bicubic/Superresolution Comparison <#write-animated-gif-with-bicubic-superresolution-comparison>`__
- `Create a Video with Sliding Bicubic/Superresolution Comparison <#create-a-video-with-sliding-bicubic-superresolution-comparison>`__
- `Superresolution on full input image <#superresolution-on-full-input-image>`__
- `Compute patches <#compute-patches>`__
- `Do Inference <#do-inference>`__
- `Save superresolution image and the bicubic image <#save-superresolution-image-and-the-bicubic-image>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Install requirements
~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
!pip install -q 'openvino>=2023.0.0'
!pip install -q "openvino>=2023.0.0"
!pip install -q opencv-python
!pip install -q pillow matplotlib
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -53,13 +89,15 @@ Imports
path.parent.mkdir(parents=True, exist_ok=True)
urllib.request.urlretrieve(url, path)
Settings
~~~~~~~~
Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Select inference device
^^^^^^^^^^^^^^^^^^^^^^^
select device from dropdown list for running inference using OpenVINO
Select inference device `⇑ <#top>`__
-------------------------------------------------------------------------------------------------------------------------------
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -107,8 +145,9 @@ select device from dropdown list for running inference using OpenVINO
else:
print(f'{model_name} already downloaded to {base_model_dir}')
Functions
~~~~~~~~~
Functions `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -168,8 +207,9 @@ Functions
"""
return cv2.cvtColor(image_data, cv2.COLOR_BGR2RGB)
Load the Superresolution Model
------------------------------
Load the Superresolution Model `⇑ <#top>`__
###############################################################################################################################
The Super Resolution model expects two inputs: the input image and a
bicubic interpolation of the input image to the target size of
@ -216,12 +256,13 @@ about the network inputs and outputs.
The image sides are upsampled by a factor of 4. The new image is 16 times as large as the original image
Load and Show the Input Image
-----------------------------
Load and Show the Input Image `⇑ <#top>`__
###############################################################################################################################
**NOTE**: For the best results, use raw images (like TIFF, BMP or
PNG). Compressed images (like JPEG) may appear distorted after
processing with the super resolution model.
**NOTE**: For the best results, use raw images (like ``TIFF``,
``BMP`` or ``PNG``). Compressed images (like ``JPEG``) may appear
distorted after processing with the super resolution model.
.. code:: ipython3
@ -251,11 +292,13 @@ Load and Show the Input Image
.. image:: 202-vision-superresolution-image-with-output_files/202-vision-superresolution-image-with-output_15_1.png
Superresolution on a Crop of the Image
--------------------------------------
Superresolution on a Crop of the Image `⇑ <#top>`__
###############################################################################################################################
Crop the Input Image once. `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Crop the Input Image once.
~~~~~~~~~~~~~~~~~~~~~~~~~~
Crop the network input size. Give the X (width) and Y (height)
coordinates for the top left corner of the crop. Set the ``CROP_FACTOR``
@ -303,8 +346,9 @@ as the crop size.
.. image:: 202-vision-superresolution-image-with-output_files/202-vision-superresolution-image-with-output_17_1.png
Reshape/Resize Crop for Model Input
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reshape/Resize Crop for Model Input `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The input image is resized to a network input size, and reshaped to
(N,C,H,W) (N=number of images, C=number of channels, H=height, W=width).
@ -326,8 +370,9 @@ interpolation. This bicubic image is the second input to the network.
input_image_original = np.expand_dims(image_crop.transpose(2, 0, 1), axis=0)
input_image_bicubic = np.expand_dims(bicubic_image.transpose(2, 0, 1), axis=0)
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Do inference and convert the inference result to an ``RGB`` image.
@ -343,8 +388,9 @@ Do inference and convert the inference result to an ``RGB`` image.
# Get inference result as numpy array and reshape to image shape and data type
result_image = convert_result_to_image(result)
Show and Save Results
~~~~~~~~~~~~~~~~~~~~~
Show and Save Results `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Show the bicubic image and the enhanced superresolution image.
@ -369,8 +415,9 @@ Show the bicubic image and the enhanced superresolution image.
.. image:: 202-vision-superresolution-image-with-output_files/202-vision-superresolution-image-with-output_23_1.png
Save Superresolution and Bicubic Image Crop
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Save Superresolution and Bicubic Image Crop `⇑ <#top>`__
-------------------------------------------------------------------------------------------------------------------------------
.. code:: ipython3
@ -402,7 +449,7 @@ Save Superresolution and Bicubic Image Crop
Write Animated GIF with Bicubic/Superresolution Comparison
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`⇑ <#top>`__
.. code:: ipython3
@ -440,7 +487,7 @@ Write Animated GIF with Bicubic/Superresolution Comparison
Create a Video with Sliding Bicubic/Superresolution Comparison
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`⇑ <#top>`__
This may take a while. For the video, the superresolution and bicubic
image are resized by a factor of 2 to improve processing speed. This
@ -507,8 +554,9 @@ the ``Files`` tool.
The video has been saved to output/flag_crop_comparison_2x.avi<br>
Superresolution on full input image
-----------------------------------
Superresolution on full input image `⇑ <#top>`__
###############################################################################################################################
Superresolution on the full image is done by dividing the image into
patches of equal size, doing superresolution on each path, and then
@ -518,8 +566,9 @@ near the border of the image are ignored.
Adjust the ``CROPLINES`` setting in the next cell if you see boundary
effects.
Compute patches
~~~~~~~~~~~~~~~
Compute patches `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -563,8 +612,9 @@ Compute patches
The output image will have a width of 11280 and a height of 7280
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The code below reads one patch of the image at a time. Each patch is
reshaped to the network input shape and upsampled with bicubic
@ -681,12 +731,13 @@ as total time to process each patch.
.. parsed-literal::
Processed 42 patches in 4.73 seconds. Total patches per second (including processing): 8.87.
Inference patches per second: 17.65
Processed 42 patches in 4.78 seconds. Total patches per second (including processing): 8.78.
Inference patches per second: 17.27
Save superresolution image and the bicubic image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Save superresolution image and the bicubic image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3

12
docs/notebooks/202-vision-superresolution-image-with-output_files/index.html
View File

@ -1,10 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/202-vision-superresolution-image-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/202-vision-superresolution-image-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/202-vision-superresolution-image-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="202-vision-superresolution-image-with-output_15_1.png">202-vision-superresolution-image-with-output_15..&gt;</a> 12-Jul-2023 00:11 272963
<a href="202-vision-superresolution-image-with-output_17_1.png">202-vision-superresolution-image-with-output_17..&gt;</a> 12-Jul-2023 00:11 356735
<a href="202-vision-superresolution-image-with-output_23_1.png">202-vision-superresolution-image-with-output_23..&gt;</a> 12-Jul-2023 00:11 2896276
<a href="202-vision-superresolution-image-with-output_27_1.png">202-vision-superresolution-image-with-output_27..&gt;</a> 12-Jul-2023 00:11 3207711
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/202-vision-superresolution-image-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="202-vision-superresolution-image-with-output_15_1.png">202-vision-superresolution-image-with-output_15..&gt;</a> 16-Aug-2023 01:31 272963
<a href="202-vision-superresolution-image-with-output_17_1.png">202-vision-superresolution-image-with-output_17..&gt;</a> 16-Aug-2023 01:31 356735
<a href="202-vision-superresolution-image-with-output_23_1.png">202-vision-superresolution-image-with-output_23..&gt;</a> 16-Aug-2023 01:31 2896276
<a href="202-vision-superresolution-image-with-output_27_1.png">202-vision-superresolution-image-with-output_27..&gt;</a> 16-Aug-2023 01:31 3207711
</pre><hr></body>
</html>

209
docs/notebooks/202-vision-superresolution-video-with-output.rst
View File

File diff suppressed because one or more lines are too long

88
docs/notebooks/203-meter-reader-with-output.rst
View File

@ -1,6 +1,8 @@
Industrial Meter Reader
=======================
.. _top:
This notebook shows how to create a industrial meter reader with
OpenVINO Runtime. We use the pre-trained
`PPYOLOv2 <https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.4/configs/ppyolo>`__
@ -19,8 +21,26 @@ to build up a multiple inference task pipeline:
workflow
Import
------
**Table of contents**:
- `Import <#import>`__
- `Prepare the Model and Test Image <#prepare-the-model-and-test-image>`__
- `Configuration <#configuration>`__
- `Load the Models <#load-the-models>`__
- `Data Process <#data-process>`__
- `Main Function <#main-function>`__
- `Initialize the model and parameters. <#initialize-the-model-and-parameters>`__
- `Run meter detection model <#run-meter-detection-model>`__
- `Run meter segmentation model <#run-meter-segmentation-model>`__
- `Postprocess the models result and calculate the final readings <#postprocess-the-models-result-and-calculate-the-final-readings>`__
- `Get the reading result on the meter picture <#get-the-reading-result-on-the-meter-picture>`__
- `Try it with your meter photos! <#try-it-with-your-meter-photos>`__
Import `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -37,18 +57,17 @@ Import
sys.path.append("../utils")
from notebook_utils import download_file, segmentation_map_to_image
Prepare the Model and Test Image
--------------------------------
Prepare the Model and Test Image `⇑ <#top>`__
###############################################################################################################################
Download PPYolov2 and DeepLabV3P pre-trained models from PaddlePaddle
community.
Download PPYOLOv2 and DeepLabV3P pre-trained models from PaddlePaddle community.
.. code:: ipython3
MODEL_DIR = "model"
DATA_DIR = "data"
DET_MODEL_LINK = "https://bj.bcebos.com/paddlex/examples2/meter_reader/meter_det_model.tar.gz"
SEG_MODEL_LINK = "https://bj.bcebos.com/paddlex/examples2/meter_reader/meter_seg_model.tar.gz"
DET_MODEL_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/meter-reader/meter_det_model.tar.gz"
SEG_MODEL_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/meter-reader/meter_seg_model.tar.gz"
DET_FILE_NAME = DET_MODEL_LINK.split("/")[-1]
SEG_FILE_NAME = SEG_MODEL_LINK.split("/")[-1]
IMG_LINK = "https://user-images.githubusercontent.com/91237924/170696219-f68699c6-1e82-46bf-aaed-8e2fc3fa5f7b.jpg"
@ -113,8 +132,8 @@ community.
Test Image Saved to "./data".
Configuration
-------------
Configuration `⇑ <#top>`__
###############################################################################################################################
Add parameter configuration for reading calculation.
@ -142,8 +161,8 @@ Add parameter configuration for reading calculation.
SEG_LABEL = {'background': 0, 'pointer': 1, 'scale': 2}
Load the Models
---------------
Load the Models `⇑ <#top>`__
###############################################################################################################################
Define a common class for model loading and inference
@ -185,8 +204,8 @@ Define a common class for model loading and inference
result = self.compiled_model(input_image)[self.output_layer]
return result
Data Process
------------
Data Process `⇑ <#top>`__
###############################################################################################################################
Including the preprocessing and postprocessing tasks of each model.
@ -515,13 +534,15 @@ Including the preprocessing and postprocessing tasks of each model.
readings.append(reading)
return readings
Main Function
-------------
Main Function `⇑ <#top>`__
###############################################################################################################################
Initialize the model and parameters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
Initialize the model and parameters. `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -581,7 +602,7 @@ bounds of input batch size.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fc73dea8c70>
<matplotlib.image.AxesImage at 0x7f41e01e70a0>
@ -589,11 +610,10 @@ bounds of input batch size.
.. image:: 203-meter-reader-with-output_files/203-meter-reader-with-output_15_1.png
Run meter detection model
~~~~~~~~~~~~~~~~~~~~~~~~~
Run meter detection model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Detect the location of the meter and prepare the ROI images for
segmentation.
Detect the location of the meter and prepare the ROI images for segmentation.
.. code:: ipython3
@ -634,8 +654,8 @@ segmentation.
.. image:: 203-meter-reader-with-output_files/203-meter-reader-with-output_17_1.png
Run meter segmentation model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run meter segmentation model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Get the results of segmentation task on detected ROI.
@ -674,10 +694,11 @@ Get the results of segmentation task on detected ROI.
.. image:: 203-meter-reader-with-output_files/203-meter-reader-with-output_19_1.png
Postprocess the models result and calculate the final readings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Postprocess the models result and calculate the final readings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use OpenCV function to find the location of the pointer in a scale map.
Use OpenCV function to find the location of the pointer in a
scale map.
.. code:: ipython3
@ -711,8 +732,9 @@ Use OpenCV function to find the location of the pointer in a scale map.
.. image:: 203-meter-reader-with-output_files/203-meter-reader-with-output_21_1.png
Get the reading result on the meter picture
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Get the reading result on the meter picture `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -742,4 +764,6 @@ Get the reading result on the meter picture
.. image:: 203-meter-reader-with-output_files/203-meter-reader-with-output_23_1.png
## Try it with your meter photos!
Try it with your meter photos! `⇑ <#top>`__
###############################################################################################################################

14
docs/notebooks/203-meter-reader-with-output_files/index.html
View File

@ -1,11 +1,11 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/203-meter-reader-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/203-meter-reader-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/203-meter-reader-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="203-meter-reader-with-output_15_1.png">203-meter-reader-with-output_15_1.png</a> 12-Jul-2023 00:11 170121
<a href="203-meter-reader-with-output_17_1.png">203-meter-reader-with-output_17_1.png</a> 12-Jul-2023 00:11 190271
<a href="203-meter-reader-with-output_19_1.png">203-meter-reader-with-output_19_1.png</a> 12-Jul-2023 00:11 26914
<a href="203-meter-reader-with-output_21_1.png">203-meter-reader-with-output_21_1.png</a> 12-Jul-2023 00:11 8966
<a href="203-meter-reader-with-output_23_1.png">203-meter-reader-with-output_23_1.png</a> 12-Jul-2023 00:11 170338
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/203-meter-reader-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="203-meter-reader-with-output_15_1.png">203-meter-reader-with-output_15_1.png</a> 16-Aug-2023 01:31 170121
<a href="203-meter-reader-with-output_17_1.png">203-meter-reader-with-output_17_1.png</a> 16-Aug-2023 01:31 190271
<a href="203-meter-reader-with-output_19_1.png">203-meter-reader-with-output_19_1.png</a> 16-Aug-2023 01:31 26914
<a href="203-meter-reader-with-output_21_1.png">203-meter-reader-with-output_21_1.png</a> 16-Aug-2023 01:31 8966
<a href="203-meter-reader-with-output_23_1.png">203-meter-reader-with-output_23_1.png</a> 16-Aug-2023 01:31 170338
</pre><hr></body>
</html>

187
docs/notebooks/204-segmenter-semantic-segmentation-with-output.rst
View File

@ -1,6 +1,8 @@
Semantic Segmentation with OpenVINO™ using Segmenter
====================================================
.. _top:
Semantic segmentation is a difficult computer vision problem with many
applications such as autonomous driving, robotics, augmented reality,
and many others. Its goal is to assign labels to each pixel according to
@ -24,7 +26,28 @@ Segmenter <https://github.com/rstrudel/segmenter>`__.
More about the model and its details can be found in the following
paper: `Segmenter: Transformer for Semantic
Segmentation <https://arxiv.org/abs/2105.05633>`__ or in the
`repository <https://github.com/rstrudel/segmenter>`__.
`repository <https://github.com/rstrudel/segmenter>`__.
**Table of contents**:
- `Get and prepare PyTorch model <#get-and-prepare-pytorch-model>`__
- `Prerequisites <#prerequisites>`__
- `Loading PyTorch model <#loading-pytorch-model>`__
- `Preparing preprocessing and visualization functions <#preparing-preprocessing-and-visualization-functions>`__
- `Preprocessing <#preprocessing>`__
- `Visualization <#visualization>`__
- `Validation of inference of original model <#validation-of-inference-of-original-model>`__
- `Export to ONNX <#export-to-onnx>`__
- `Convert ONNX model to OpenVINO Intermediate Representation (IR) <#convert-onnx-model-to-openvino-intermediate-representation-ir>`__
- `Verify converted model inference <#verify-converted-model-inference>`__
- `Select inference device <#select-inference-device>`__
- `Benchmarking performance of converted model <#benchmarking-performance-of-converted-model>`__
.. |Segmenteer diagram| image:: https://user-images.githubusercontent.com/24582831/148507554-87eb80bd-02c7-4c31-b102-c6141e231ec8.png
@ -39,13 +62,14 @@ notebook consists of the following steps:
- Validating inference of the converted model
- Benchmark performance of the converted model
Get and prepare PyTorch model
-----------------------------
Get and prepare PyTorch model `⇑ <#top>`__
###############################################################################################################################
The first thing we’ll need to do is clone
`repository <https://github.com/rstrudel/segmenter>`__ containing model
and helper functions. We will use Tiny model with mask transformer, that
is Seg-T-Mask/16. There are also better, but much larger models
is ``Seg-T-Mask/16``. There are also better, but much larger models
available in the linked repo. This model is pre-trained on
`ADE20K <https://groups.csail.mit.edu/vision/datasets/ADE20K/>`__
dataset used for segmentation.
@ -54,8 +78,9 @@ The code from the repository already contains functions that create
model and load weights, but we will need to download config and trained
weights (checkpoint) file and add some additional helper functions.
Prerequisites
~~~~~~~~~~~~~
Prerequisites `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -86,8 +111,8 @@ Prerequisites
)
from notebook_utils import download_file, load_image
We’ll need timm, mmsegmentation, enops and mmcv, to use functions from
segmenter repo
We’ll need ``timm``, ``mmsegmentation``, ``einops`` and ``mmcv``, to use
functions from segmenter repo
First, we will clone the Segmenter repo and then download weights and
config for our model.
@ -109,7 +134,7 @@ config for our model.
Cloning into 'segmenter'...
remote: Enumerating objects: 268, done.
remote: Total 268 (delta 0), reused 0 (delta 0), pack-reused 268
Receiving objects: 100% (268/268), 15.34 MiB | 4.21 MiB/s, done.
Receiving objects: 100% (268/268), 15.34 MiB | 3.75 MiB/s, done.
Resolving deltas: 100% (117/117), done.
@ -117,8 +142,8 @@ config for our model.
# download config and pretrained model weights
# here we use tiny model, there are also better but larger models available in repository
WEIGHTS_LINK = "https://www.rocq.inria.fr/cluster-willow/rstrudel/segmenter/checkpoints/ade20k/seg_tiny_mask/checkpoint.pth"
CONFIG_LINK = "https://www.rocq.inria.fr/cluster-willow/rstrudel/segmenter/checkpoints/ade20k/seg_tiny_mask/variant.yml"
WEIGHTS_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/segmenter/checkpoints/ade20k/seg_tiny_mask/checkpoint.pth"
CONFIG_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/segmenter/checkpoints/ade20k/seg_tiny_mask/variant.yml"
MODEL_DIR = Path("model/")
MODEL_DIR.mkdir(exist_ok=True)
@ -142,8 +167,9 @@ config for our model.
model/variant.yml: 0%| | 0.00/940 [00:00<?, ?B/s]
Loading PyTorch model
~~~~~~~~~~~~~~~~~~~~~
Loading PyTorch model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PyTorch models are usually an instance of
`torch.nn.Module <https://pytorch.org/docs/stable/generated/torch.nn.Module.html>`__
@ -187,26 +213,29 @@ Load normalization settings from config file.
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
Preparing preprocessing and visualization functions
---------------------------------------------------
Preparing preprocessing and visualization functions `⇑ <#top>`__
###############################################################################################################################
Now we will define utility functions for preprocessing and visualizing
the results.
Preprocessing
~~~~~~~~~~~~~
Preprocessing `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Inference input is tensor with shape ``[1, 3, H, W]`` in ``B, C, H, W``
format, where:
* ``B`` - batch size (in our case 1, as we are just adding 1 with unsqueeze)
* ``C`` - image channels (in our case RGB - 3)
* ``H`` - image height
* ``W`` - image width
- ``B`` - batch size (in our case 1, as we are just adding 1 with
unsqueeze)
- ``C`` - image channels (in our case RGB - 3)
- ``H`` - image height
- ``W`` - image width
Resizing to the correct scale and splitting to batches is done inside
inference, so we don’t need to resize or split the image in
@ -241,15 +270,16 @@ normalized with given mean and standard deviation provided in
return im
Visualization
~~~~~~~~~~~~~
Visualization `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Inference output contains labels assigned to each pixel, so the output
in our case is ``[150, H, W]`` in ``CL, H, W`` format where:
in our case is ``[150, H, W]`` in ``CL, H, W`` format where:
* ``CL`` - number of classes for labels (in our case 150)
* ``H`` - image height
* ``W`` - image width
- ``CL`` - number of classes for labels (in our case 150)
- ``H`` - image height
- ``W`` - image width
Since we want to visualize this output, we reduce dimensions to
``[1, H, W]`` where we keep only class with the highest value as that is
@ -285,8 +315,9 @@ corresponding to the inferred labels.
return pil_blend
Validation of inference of original model
-----------------------------------------
Validation of inference of original model `⇑ <#top>`__
###############################################################################################################################
Now that we have everything ready, we can perform segmentation on
example image ``coco_hollywood.jpg``.
@ -337,8 +368,9 @@ We can see that model segments the image into meaningful parts. Since we
are using tiny variant of model, the result is not as good as it is with
larger models, but it already shows nice segmentation performance.
Export to ONNX
--------------
Export to ONNX `⇑ <#top>`__
###############################################################################################################################
Now that we’ve verified that the inference of PyTorch model works, we
will first export it to ONNX format.
@ -347,15 +379,16 @@ To do this, we first get input dimensions from the model configuration
file and create torch dummy input. Input dimensions are in our case
``[2, 3, 512, 512]`` in ``B, C, H, W]`` format, where:
* ``B`` - batch size
* ``C`` - image channels (in our case RGB - 3)
* ``H`` - model input image height
* ``W`` - model input image width
- ``B`` - batch size
- ``C`` - image channels (in our case RGB - 3)
- ``H`` - model input image height
- ``W`` - model input image width
.. note::
..
Note that H and W are here fixed to 512, as this is required by the model. Resizing is
done inside the inference function from the original repository.
Note that H and W are here fixed to 512, as this is required by the
model. Resizing is done inside the inference function from the
original repository.
After that, we use ``export`` function from PyTorch to convert the model
to ONNX. The process can generate some warnings, but they are not a
@ -388,36 +421,37 @@ problem.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:69: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:69: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if H % patch_size > 0:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:71: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:71: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if W % patch_size > 0:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/vit.py:122: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/vit.py:122: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x.shape[1] != pos_embed.shape[1]:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/decoder.py:100: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/decoder.py:100: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
masks = rearrange(masks, "b (h w) n -> b n h w", h=int(GS))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if extra_h > 0:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/notebooks/204-segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if extra_w > 0:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py:258: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py:258: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
Convert ONNX model to OpenVINO Intermediate Representation (IR)
---------------------------------------------------------------
Convert ONNX model to OpenVINO Intermediate Representation (IR). `⇑ <#top>`__
###############################################################################################################################
While ONNX models are directly supported by OpenVINO runtime, it can be
useful to convert them to IR format to take advantage of OpenVINO
optimization tools and features. ``mo.convert_model`` function can be
used for converting model using OpenVINO Model Optimizer. The function
returns instance of OpenVINO Model class, which is ready to use in
Python interface but can also be serialized to OpenVINO IR format for
future execution.
optimization tools and features. The ``mo.convert_model`` function of
`model conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
can be used. The function returns instance of OpenVINO Model class,
which is ready to use in Python interface but can also be serialized to
OpenVINO IR format for future execution.
.. code:: ipython3
@ -428,8 +462,9 @@ future execution.
# serialize model for saving IR
serialize(model, str(MODEL_DIR / "segmenter.xml"))
Verify converted model inference
--------------------------------
Verify converted model inference `⇑ <#top>`__
###############################################################################################################################
To test that model was successfully converted, we can use same inference
function from original repository, but we need to make custom class.
@ -496,13 +531,14 @@ any additional custom code required to process input.
"""
return torch.from_numpy(self.model(data)[self.output_blob])
Now that we have created SegmenterOV helper class, we can use it in
Now that we have created ``SegmenterOV`` helper class, we can use it in
inference function.
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
select device from dropdown list for running inference using OpenVINO
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
@ -560,8 +596,9 @@ select device from dropdown list for running inference using OpenVINO
As we can see, we get the same results as with original model.
Benchmarking performance of converted model
-------------------------------------------
Benchmarking performance of converted model `⇑ <#top>`__
###############################################################################################################################
Finally, use the OpenVINO `Benchmark
Tool <https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html>`__
@ -606,18 +643,18 @@ to measure the inference performance of the model.
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] Build ................................. 2023.0.1-11005-fa1c41994f3-releases/2023/0
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] Build ................................. 2023.0.1-11005-fa1c41994f3-releases/2023/0
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 16.16 ms
[ INFO ] Read model took 16.30 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input (node: input) : f32 / [...] / [2,3,512,512]
@ -631,7 +668,7 @@ to measure the inference performance of the model.
[ INFO ] Model outputs:
[ INFO ] output (node: output) : f32 / [...] / [2,150,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 349.84 ms
[ INFO ] Compile model took 343.28 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
@ -660,15 +697,15 @@ to measure the inference performance of the model.
[ INFO ] Fill input 'input' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 219.27 ms
[ INFO ] First inference took 227.83 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1374 iterations
[ INFO ] Duration: 120364.60 ms
[ INFO ] Count: 1332 iterations
[ INFO ] Duration: 120630.17 ms
[ INFO ] Latency:
[ INFO ] Median: 523.66 ms
[ INFO ] Average: 524.91 ms
[ INFO ] Min: 217.86 ms
[ INFO ] Max: 612.22 ms
[ INFO ] Throughput: 22.83 FPS
[ INFO ] Median: 542.28 ms
[ INFO ] Average: 542.75 ms
[ INFO ] Min: 344.15 ms
[ INFO ] Max: 609.17 ms
[ INFO ] Throughput: 22.08 FPS

12
docs/notebooks/204-segmenter-semantic-segmentation-with-output_files/index.html
View File

@ -1,10 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/204-segmenter-semantic-segmentation-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/204-segmenter-semantic-segmentation-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/204-segmenter-semantic-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="204-segmenter-semantic-segmentation-with-output_21_0.jpg">204-segmenter-semantic-segmentation-with-output..&gt;</a> 12-Jul-2023 00:11 72352
<a href="204-segmenter-semantic-segmentation-with-output_21_0.png">204-segmenter-semantic-segmentation-with-output..&gt;</a> 12-Jul-2023 00:11 909669
<a href="204-segmenter-semantic-segmentation-with-output_34_0.jpg">204-segmenter-semantic-segmentation-with-output..&gt;</a> 12-Jul-2023 00:11 72356
<a href="204-segmenter-semantic-segmentation-with-output_34_0.png">204-segmenter-semantic-segmentation-with-output..&gt;</a> 12-Jul-2023 00:11 909691
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/204-segmenter-semantic-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="204-segmenter-semantic-segmentation-with-output_21_0.jpg">204-segmenter-semantic-segmentation-with-output..&gt;</a> 16-Aug-2023 01:31 72352
<a href="204-segmenter-semantic-segmentation-with-output_21_0.png">204-segmenter-semantic-segmentation-with-output..&gt;</a> 16-Aug-2023 01:31 909669
<a href="204-segmenter-semantic-segmentation-with-output_34_0.jpg">204-segmenter-semantic-segmentation-with-output..&gt;</a> 16-Aug-2023 01:31 72356
<a href="204-segmenter-semantic-segmentation-with-output_34_0.png">204-segmenter-semantic-segmentation-with-output..&gt;</a> 16-Aug-2023 01:31 909691
</pre><hr></body>
</html>

161
docs/notebooks/205-vision-background-removal-with-output.rst
View File

@ -1,24 +1,51 @@
Image Background Removal with U^2-Net and OpenVINO™
===================================================
.. _top:
This notebook demonstrates background removal in images using
U\ :math:`^2`-Net and OpenVINO.
For more information about U\ :math:`^2`-Net, including source code and
test data, see the `Github
test data, see the `GitHub
page <https://github.com/xuebinqin/U-2-Net>`__ and the research paper:
`U^2-Net: Going Deeper with Nested U-Structure for Salient Object
Detection <https://arxiv.org/pdf/2005.09007.pdf>`__.
The PyTorch U\ :math:`^2`-Net model is converted to OpenVINO IR format.
The model source is available
`here <https://github.com/xuebinqin/U-2-Net>`__.
`here <https://github.com/xuebinqin/U-2-Net>`__.
Preparation
-----------
Install requirements
~~~~~~~~~~~~~~~~~~~~
**Table of contents**:
- `Preparation <#preparation>`__
- `Install requirements <#install-requirements>`__
- `Import the PyTorch Library and U2-Net <#import-the-pytorch-library-and-u2-net>`__
- `Settings <#settings>`__
- `Load the U2-Net Model <#load-the-u2-net-model>`__
- `Convert PyTorch U2-Net model to OpenVINO IR <#convert-pytorch-u2-net-model-to-openvino-ir>`__
- `Convert Pytorch model to OpenVINO IR Format <#convert-pytorch-model-to-openvino-ir-format>`__
- `Load and Pre-Process Input Image <#load-and-pre-process-input-image>`__
- `Select inference device <#select-inference-device>`__
- `Do Inference on OpenVINO IR Model <#do-inference-on-openvino-ir-model>`__
- `Visualize Results <#visualize-results>`__
- `Add a Background Image <#add-a-background-image>`__
- `References <#references>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -26,8 +53,9 @@ Install requirements
!pip install -q torch onnx opencv-python matplotlib
!pip install -q gdown
Import the PyTorch Library and U\ :math:`^2`-Net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Import the PyTorch Library and U\ :math:`^2`-Net `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -63,8 +91,9 @@ Import the PyTorch Library and U\ :math:`^2`-Net
from notebook_utils import load_image
from model.u2net import U2NET, U2NETP
Settings
~~~~~~~~
Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
This tutorial supports using the original U\ :math:`^2`-Net salient
object detection model, as well as the smaller U2NETP version. Two sets
@ -103,8 +132,9 @@ detection and human segmentation.
MODEL_DIR = "model"
model_path = Path(MODEL_DIR) / u2net_model.name / Path(u2net_model.name).with_suffix(".pth")
Load the U\ :math:`^2`-Net Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the U\ :math:`^2`-Net Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The U\ :math:`^2`-Net human segmentation model weights are stored on
Google Drive. They will be downloaded if they are not present yet. The
@ -132,8 +162,7 @@ next cell loads the model and the pre-trained weights.
Downloading...
From: https://drive.google.com/uc?id=1rbSTGKAE-MTxBYHd-51l2hMOQPT_7EPy
To: <_io.BufferedWriter name='model/u2net_lite/u2net_lite.pth'>
100%|██████████| 4.68M/4.68M [00:00<00:00, 4.90MB/s]
100%|██████████| 4.68M/4.68M [00:01<00:00, 3.98MB/s]
.. parsed-literal::
@ -160,37 +189,39 @@ next cell loads the model and the pre-trained weights.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/functional.py:3734: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/functional.py:3734: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/functional.py:1967: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/functional.py:1967: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py:258: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py:258: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at ../torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1884.)
_C._jit_pass_onnx_graph_shape_type_inference(
Convert PyTorch U\ :math:`^2`-Net model to OpenVINO IR
------------------------------------------------------
Convert PyTorch U\ :math:`^2`-Net model to OpenVINO IR `⇑ <#top>`__
###############################################################################################################################
Convert Pytorch model to OpenVINO IR Format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use Model Optimizer Python API to convert the Pytorch model to OpenVINO
IR format, with ``FP16`` precision. We add the mean values to the model
and scale the input with the standard deviation with ``scale_values``
parameter. With these options, it is not necessary to normalize input
data before propagating it through the network. The mean and standard
deviation values can be found in the
Convert Pytorch model to OpenVINO IR Format `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To convert the Pytorch model to OpenVINO IR format with ``FP16``
precision, use model conversion Python API . We add the mean values to
the model and scale the input with the standard deviation with
``scale_values`` parameter. With these options, it is not necessary to
normalize input data before propagating it through the network. The mean
and standard deviation values can be found in the
`dataloader <https://github.com/xuebinqin/U-2-Net/blob/master/data_loader.py>`__
file in the `U^2-Net
repository <https://github.com/xuebinqin/U-2-Net/>`__ and multiplied by
255 to support images with pixel values from 0-255.
For more information, refer to the `Model Optimizer Developer
Guide <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
For more information about model conversion, refer to this
`page <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
Executing the following command may take a while.
@ -203,8 +234,9 @@ Executing the following command may take a while.
compress_to_fp16=True
)
Load and Pre-Process Input Image
--------------------------------
Load and Pre-Process Input Image `⇑ <#top>`__
###############################################################################################################################
While OpenCV reads images in ``BGR`` format, the OpenVINO IR model
expects images in ``RGB``. Therefore, convert the images to ``RGB``,
@ -224,16 +256,46 @@ that is expected by the OpenVINO IR model.
# for OpenVINO IR model: (1, 3, 512, 512).
input_image = np.expand_dims(np.transpose(resized_image, (2, 0, 1)), 0)
Do Inference on OpenVINO IR Model
---------------------------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
import ipywidgets as widgets
core = Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
Do Inference on OpenVINO IR Model `⇑ <#top>`__
###############################################################################################################################
Load the OpenVINO IR model to OpenVINO Runtime and do inference.
.. code:: ipython3
# Load the network to OpenVINO Runtime.
ie = Core()
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")
core = Core()
compiled_model_ir = core.compile_model(model=model_ir, device_name=device.value)
# Get the names of input and output layers.
input_layer_ir = compiled_model_ir.input(0)
output_layer_ir = compiled_model_ir.output(0)
@ -250,11 +312,12 @@ Load the OpenVINO IR model to OpenVINO Runtime and do inference.
.. parsed-literal::
Inference finished. Inference time: 0.122 seconds, FPS: 8.17.
Inference finished. Inference time: 0.119 seconds, FPS: 8.43.
Visualize Results
-----------------
Visualize Results `⇑ <#top>`__
###############################################################################################################################
Show the original image, the segmentation result, and the original image
with the background removed.
@ -281,11 +344,12 @@ with the background removed.
.. image:: 205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_20_0.png
.. image:: 205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_22_0.png
Add a Background Image
~~~~~~~~~~~~~~~~~~~~~~
Add a Background Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In the segmentation result, all foreground pixels have a value of 1, all
background pixels a value of 0. Replace the background image as follows:
@ -340,7 +404,7 @@ background pixels a value of 0. Replace the background image as follows:
.. image:: 205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_22_0.png
.. image:: 205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_24_0.png
@ -349,13 +413,14 @@ background pixels a value of 0. Replace the background image as follows:
The generated image <code>coco_hollywood-wall.jpg</code> is saved in the directory <code>output</code>. You can also download the image by clicking on this link: output/coco_hollywood-wall.jpg<br>
References
----------
References `⇑ <#top>`__
###############################################################################################################################
- `PIP install
openvino-dev <https://github.com/openvinotoolkit/openvino/blob/releases/2021/3/docs/install_guides/pypi-openvino-dev.md>`__
- `Model Optimizer
Documentation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__
- `Model Conversion
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
- `U^2-Net <https://github.com/xuebinqin/U-2-Net>`__
- U^2-Net research paper: `U^2-Net: Going Deeper with Nested
U-Structure for Salient Object

3
docs/notebooks/205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_20_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:83192799a2c31d56beff242e7d9b1dae795e6cb3ef9c6f754c8cdc102f3af8dd
size 279567

4
docs/notebooks/205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_22_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a63381a3ccf5d7a134519b437c3197abb82df80edc51bc8fa51c574a186ac1cc
size 927148
oid sha256:83192799a2c31d56beff242e7d9b1dae795e6cb3ef9c6f754c8cdc102f3af8dd
size 279567

3
docs/notebooks/205-vision-background-removal-with-output_files/205-vision-background-removal-with-output_24_0.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a63381a3ccf5d7a134519b437c3197abb82df80edc51bc8fa51c574a186ac1cc
size 927148

8
docs/notebooks/205-vision-background-removal-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/205-vision-background-removal-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/205-vision-background-removal-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/205-vision-background-removal-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="205-vision-background-removal-with-output_20_0.png">205-vision-background-removal-with-output_20_0.png</a> 12-Jul-2023 00:11 279567
<a href="205-vision-background-removal-with-output_22_0.png">205-vision-background-removal-with-output_22_0.png</a> 12-Jul-2023 00:11 927148
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/205-vision-background-removal-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="205-vision-background-removal-with-output_22_0.png">205-vision-background-removal-with-output_22_0.png</a> 16-Aug-2023 01:31 279567
<a href="205-vision-background-removal-with-output_24_0.png">205-vision-background-removal-with-output_24_0.png</a> 16-Aug-2023 01:31 927148
</pre><hr></body>
</html>

193
docs/notebooks/206-vision-paddlegan-anime-with-output.rst
View File

@ -1,6 +1,8 @@
Photos to Anime with PaddleGAN and OpenVINO
===========================================
.. _top:
This tutorial demonstrates converting a
`PaddlePaddle/PaddleGAN <https://github.com/PaddlePaddle/PaddleGAN>`__
AnimeGAN model to OpenVINO IR format, and shows inference results on the
@ -14,17 +16,45 @@ documentation <https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US
anime
Preparation
-----------
**Table of contents**:
Install requirements
~~~~~~~~~~~~~~~~~~~~
- `Preparation <#preparation>`__
- `Install requirements <#install-requirements>`__
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Functions <#functions>`__
- `Inference on PaddleGAN Model <#inference-on-paddlegan-model>`__
- `Show Inference Results on PaddleGAN model <#show-inference-results-on-paddlegan-model>`__
- `Model Conversion to ONNX and OpenVINO IR <#model-conversion-to-onnx-and-openvino-ir>`__
- `Convert to ONNX <#convert-to-onnx>`__
- `Convert to OpenVINO IR <#convert-to-openvino-ir>`__
- `Show Inference Results on OpenVINO IR and PaddleGAN Models <#show-inference-results-on-openvino-ir-and-paddlegan-models>`__
- `Create Postprocessing Functions <#create-postprocessing-functions>`__
- `Do Inference on OpenVINO IR Model <#do-inference-on-openvino-ir-model>`__
- `Select inference device <#select-inference-device>`__
- `Performance Comparison <#performance-comparison>`__
- `References <#references>`__
Preparation `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
!pip install -q "openvino-dev>=2023.0.0"
!pip install -q "paddlepaddle==2.5.0rc0" "paddle2onnx>=0.6"
!pip install -q "paddlepaddle==2.5.0" "paddle2onnx>=0.6"
!pip install -q "git+https://github.com/PaddlePaddle/PaddleGAN.git" --no-deps
!pip install -q opencv-python matplotlib scikit-learn scikit-image
@ -36,13 +66,13 @@ Install requirements
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
paddleclas 2.5.1 requires faiss-cpu==1.7.1.post2, but you have faiss-cpu 1.7.4 which is incompatible.
paddleclas 2.5.1 requires gast==0.3.3, but you have gast 0.4.0 which is incompatible.
ppgan 2.1.0 requires librosa==0.8.1, but you have librosa 0.10.0.post2 which is incompatible.
ppgan 2.1.0 requires opencv-python<=4.6.0.66, but you have opencv-python 4.8.0.74 which is incompatible.
ppgan 2.1.0 requires librosa==0.8.1, but you have librosa 0.10.1 which is incompatible.
ppgan 2.1.0 requires opencv-python<=4.6.0.66, but you have opencv-python 4.8.0.76 which is incompatible.
scikit-image 0.21.0 requires imageio>=2.27, but you have imageio 2.9.0 which is incompatible.
Imports
~~~~~~~
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -85,8 +115,8 @@ Imports
)
raise
Settings
~~~~~~~~
Settings `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -100,8 +130,8 @@ Settings
ir_path = model_path.with_suffix(".xml")
onnx_path = model_path.with_suffix(".onnx")
Functions
~~~~~~~~~
Functions `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -115,8 +145,8 @@ Functions
image = cv2.resize(image, (max_width, new_height))
return image
Inference on PaddleGAN Model
----------------------------
Inference on PaddleGAN Model `⇑ <#top>`__
###############################################################################################################################
The PaddleGAN
`documentation <https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/tutorials/animegan.md>`__
@ -133,7 +163,7 @@ source of the function.
.. parsed-literal::
[07/11 23:03:15] ppgan INFO: Found /opt/home/k8sworker/.cache/ppgan/animeganv2_hayao.pdparams
[08/17 16:13:48] ppgan INFO: Found /opt/home/k8sworker/.cache/ppgan/animeganv2_hayao.pdparams
.. code:: ipython3
@ -211,8 +241,8 @@ cell.
The anime image was saved to output/coco_bricks_anime_pg.jpg
Show Inference Results on PaddleGAN model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Show Inference Results on PaddleGAN model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -228,15 +258,15 @@ Show Inference Results on PaddleGAN model
.. image:: 206-vision-paddlegan-anime-with-output_files/206-vision-paddlegan-anime-with-output_15_0.png
Model Conversion to ONNX and OpenVINO IR
----------------------------------------
Model Conversion to ONNX and OpenVINO IR `⇑ <#top>`__
###############################################################################################################################
Convert the PaddleGAN model to OpenVINO IR by first converting PaddleGAN
to ONNX with ``paddle2onnx`` and then converting the ONNX model to
OpenVINO IR with Model Optimizer.
OpenVINO IR with model conversion API.
Convert to ONNX
~~~~~~~~~~~~~~~
Convert to ONNX `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Exporting to ONNX requires specifying an input shape with PaddlePaddle
``InputSpec`` and calling ``paddle.onnx.export``. Then, check the input
@ -268,23 +298,23 @@ succeeds, the output of the next cell will include
.. parsed-literal::
2023-07-11 23:03:23 [INFO] Static PaddlePaddle model saved in model/paddle_model_static_onnx_temp_dir.
2023-08-17 16:13:56 [INFO] Static PaddlePaddle model saved in model/paddle_model_static_onnx_temp_dir.
[Paddle2ONNX] Start to parse PaddlePaddle model...
[Paddle2ONNX] Model file path: model/paddle_model_static_onnx_temp_dir/model.pdmodel
[Paddle2ONNX] Paramters file path: model/paddle_model_static_onnx_temp_dir/model.pdiparams
[Paddle2ONNX] Start to parsing Paddle model...
[Paddle2ONNX] Use opset_version = 11 for ONNX export.
[Paddle2ONNX] PaddlePaddle model is exported as ONNX format now.
2023-07-11 23:03:24 [INFO] ONNX model saved in model/paddlegan_anime.onnx.
2023-08-17 16:13:56 [INFO] ONNX model saved in model/paddlegan_anime.onnx.
.. parsed-literal::
I0711 23:03:23.947630 3455115 interpretercore.cc:267] New Executor is Running.
I0817 16:13:56.664121 2277406 interpretercore.cc:237] New Executor is Running.
Convert to OpenVINO IR
~~~~~~~~~~~~~~~~~~~~~~
Convert to OpenVINO IR `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The OpenVINO IR format enables storing the preprocessing normalization
in the model file. It is then no longer necessary to normalize input
@ -319,19 +349,18 @@ normalize uses a mean and scale of ``[127.5, 127.5, 127.5]``.
The ``ResizeToScale`` class is called with ``(256,256)`` as the argument
for size. Further analysis shows that this is the minimum size to resize
to. The ``ResizeToScale`` class transform resizes images to the size
specified in the ``ResizeToScale`` params, with width and height as
specified in the ``ResizeToScale`` parameters, with width and height as
multiples of 32.
Once the mean and standard deviation values, and the shape of the model
inputs are known, you can use Model Optimizer and convert the model to
OpenVINO IR with these values. Use ``FP16`` precision and set log level
to ``CRITICAL`` to ignore warnings that are irrelevant for this demo.
For information about setting the parameters, see the `Model Optimizer
Documentation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__
.
inputs are known, you can use model conversion API and convert the model
to OpenVINO IR with these values. Use ``FP16`` precision and set log
level to ``CRITICAL`` to ignore warnings that are irrelevant for this
demo. For information about setting the parameters, see this
`page <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
**Convert ONNX Model to OpenVINO IR with**\ `Model Optimizer Python
API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
**Convert ONNX Model to OpenVINO IR with**\ `Model Conversion Python
API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
.. code:: ipython3
@ -357,19 +386,20 @@ API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
Exporting ONNX model to OpenVINO IR... This may take a few minutes.
Show Inference Results on OpenVINO IR and PaddleGAN Models
----------------------------------------------------------
Show Inference Results on OpenVINO IR and PaddleGAN Models `⇑ <#top>`__
###############################################################################################################################
If the output of Model Optimizer in the cell above showed *SUCCESS*, the
model conversion succeeded and the OpenVINO IR model has been generated.
If the conversion is successful, the output of model conversion API in
the cell above will show *SUCCESS*, and the OpenVINO IR model will be
generated.
Now, use the model for inference with the ``adjust_brightness()`` method
from the PaddleGAN model. However, in order to use the OpenVINO IR model
without installing PaddleGAN, it is useful to check what these functions
do and extract them.
Create Postprocessing Functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create Postprocessing Functions `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -412,8 +442,8 @@ OpenVINO IR model
dstf = np.uint8(dstf)
return dstf
Do Inference on OpenVINO IR Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Do Inference on OpenVINO IR Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load the OpenVINO IR model and do inference, following the same steps as
for the PaddleGAN model. For more information about inference on
@ -424,12 +454,40 @@ The OpenVINO IR model is generated with an input shape that is computed
based on the input image. If you do inference on images with different
input shapes, results may differ from the PaddleGAN results.
Select inference device `⇑ <#top>`__
-------------------------------------------------------------------------------------------------------------------------------
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
import ipywidgets as widgets
core = Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
.. code:: ipython3
# Load and prepare the IR model.
ie = Core()
model = ie.read_model(model=ir_path)
compiled_model = ie.compile_model(model=model, device_name="CPU")
core = Core()
model = core.read_model(model=ir_path)
compiled_model = core.compile_model(model=model, device_name=device.value)
input_key = compiled_model.input(0)
output_key = compiled_model.output(0)
@ -480,11 +538,12 @@ input shapes, results may differ from the PaddleGAN results.
.. image:: 206-vision-paddlegan-anime-with-output_files/206-vision-paddlegan-anime-with-output_36_0.png
.. image:: 206-vision-paddlegan-anime-with-output_files/206-vision-paddlegan-anime-with-output_37_0.png
Performance Comparison
----------------------
Performance Comparison `⇑ <#top>`__
###############################################################################################################################
Measure the time it takes to do inference on an image. This gives an
indication of performance. It is not a perfect measure. Since the
@ -505,24 +564,6 @@ measure inference on one image. For more accurate benchmarking, use
f"seconds per image, FPS: {NUM_IMAGES/time_ir:.2f}"
)
## Uncomment the lines below to measure inference time on an Intel iGPU.
## Note that it will take some time to load the model to GPU.
# If "GPU" in ie.available_devices:
# # Loading the IR model on GPU takes some time.
# compiled_model = ie.compile_model(model=model, device_name="GPU")
# start = time.perf_counter()
# for _ in range(NUM_IMAGES):
# exec_net_multi([input_image])
# end = time.perf_counter()
# time_ir = end - start
# print(
# f"OpenVINO IR model in OpenVINO Runtime/GPU: {time_ir/NUM_IMAGES:.3f} "
# f"seconds per image, FPS: {NUM_IMAGES/time_ir:.2f}"
# )
# else:
# print("A supported iGPU device is not available on this system.")
## `PADDLEGAN_INFERENCE` is defined in the "Inference on PaddleGAN model" section above.
## Uncomment the next line to enable a performance comparison with the PaddleGAN model
## if you disabled it earlier.
@ -544,19 +585,17 @@ measure inference on one image. For more accurate benchmarking, use
.. parsed-literal::
OpenVINO IR model in OpenVINO Runtime/CPU: 0.473 seconds per image, FPS: 2.12
PaddleGAN model on CPU: 6.541 seconds per image, FPS: 0.15
OpenVINO IR model in OpenVINO Runtime/CPU: 0.469 seconds per image, FPS: 2.13
PaddleGAN model on CPU: 6.121 seconds per image, FPS: 0.16
References
----------
References `⇑ <#top>`__
###############################################################################################################################
- `PaddleGAN <https://github.com/PaddlePaddle/PaddleGAN>`__
- `Paddle2ONNX <https://github.com/PaddlePaddle/paddle2onnx>`__
- `OpenVINO ONNX
support <https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_ONNX_Support.html>`__
- `OpenVINO Model Optimizer
Documentation <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__
- `OpenVINO ONNX support <https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_ONNX_Support.html>`__
- `Model Conversion API <https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html>`__
The PaddleGAN code that is shown in this notebook is written by
PaddlePaddle Authors and licensed under the Apache 2.0 license. The

0
docs/notebooks/206-vision-paddlegan-anime-with-output_files/206-vision-paddlegan-anime-with-output_36_0.png → docs/notebooks/206-vision-paddlegan-anime-with-output_files/206-vision-paddlegan-anime-with-output_37_0.png
View File

8
docs/notebooks/206-vision-paddlegan-anime-with-output_files/index.html
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/206-vision-paddlegan-anime-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/206-vision-paddlegan-anime-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/206-vision-paddlegan-anime-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="206-vision-paddlegan-anime-with-output_15_0.png">206-vision-paddlegan-anime-with-output_15_0.png</a> 12-Jul-2023 00:11 1810982
<a href="206-vision-paddlegan-anime-with-output_36_0.png">206-vision-paddlegan-anime-with-output_36_0.png</a> 12-Jul-2023 00:11 1931653
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/206-vision-paddlegan-anime-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="206-vision-paddlegan-anime-with-output_15_0.png">206-vision-paddlegan-anime-with-output_15_0.png</a> 16-Aug-2023 01:31 1810982
<a href="206-vision-paddlegan-anime-with-output_37_0.png">206-vision-paddlegan-anime-with-output_37_0.png</a> 16-Aug-2023 01:31 1931653
</pre><hr></body>
</html>

132
docs/notebooks/207-vision-paddlegan-superresolution-with-output.rst
View File

@ -1,6 +1,8 @@
Super Resolution with PaddleGAN and OpenVINO™
=============================================
.. _top:
This notebook demonstrates converting the RealSR (real-world
super-resolution) model from
`PaddlePaddle/PaddleGAN <https://github.com/PaddlePaddle/PaddleGAN>`__
@ -16,8 +18,29 @@ from CVPR 2020.
This notebook works best with small images (up to 800x600 resolution).
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Inference on PaddlePaddle Model <#inference-on-paddlepaddle-model>`__
- `Investigate PaddleGAN Model <#investigate-paddlegan-model>`__
- `Do Inference <#do-inference>`__
- `Convert PaddleGAN Model to ONNX and OpenVINO IR <#convert-paddlegan-model-to-onnx-and-openvino-ir>`__
- `Convert PaddlePaddle Model to ONNX <#convert-paddlepaddle-model-to-onnx>`__
- `Convert ONNX Model to OpenVINO IR with Model Conversion Python API <#convert-onnx-model-to-openvino-ir-with-model-conversion-python-api>`__
- `Do Inference on OpenVINO IR Model <#do-inference-on-openvino-ir-model>`__
- `Select inference device <#select-inference-device>`__
- `Show an Animated GIF <#show-an-animated-gif>`__
- `Create a Comparison Video <#create-a-comparison-video>`__
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -49,8 +72,9 @@ Imports
sys.path.append("../utils")
from notebook_utils import NotebookAlert
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -64,11 +88,13 @@ Settings
ir_path = model_path.with_suffix(".xml")
onnx_path = model_path.with_suffix(".onnx")
Inference on PaddlePaddle Model
-------------------------------
Inference on PaddlePaddle Model `⇑ <#top>`__
###############################################################################################################################
Investigate PaddleGAN Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Investigate PaddleGAN Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `PaddleGAN
documentation <https://github.com/PaddlePaddle/PaddleGAN>`__ explains
@ -86,7 +112,7 @@ source code.
.. parsed-literal::
[07/11 23:03:50] ppgan INFO: Found /opt/home/k8sworker/.cache/ppgan/DF2K_JPEG.pdparams
[08/15 23:08:25] ppgan INFO: Found /opt/home/k8sworker/.cache/ppgan/DF2K_JPEG.pdparams
.. code:: ipython3
@ -124,8 +150,9 @@ To get more information about how the model looks like, use the
# sr.model??
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To show inference on the PaddlePaddle model, set ``PADDLEGAN_INFERENCE``
to ``True`` in the cell below. Keep in mind that performing inference
@ -169,15 +196,17 @@ may take some time.
print(f"Inference duration: {duration:.2f} seconds")
plt.imshow(result_image);
Convert PaddleGAN Model to ONNX and OpenVINO IR
-----------------------------------------------
Convert PaddleGAN Model to ONNX and OpenVINO IR `⇑ <#top>`__
###############################################################################################################################
To convert the PaddlePaddle model to OpenVINO IR, first convert the
model to ONNX, and then convert the ONNX model to the OpenVINO IR
format.
Convert PaddlePaddle Model to ONNX
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Convert PaddlePaddle Model to ONNX `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -194,12 +223,12 @@ Convert PaddlePaddle Model to ONNX
.. parsed-literal::
2023-07-11 23:03:56 [INFO] Static PaddlePaddle model saved in model/paddle_model_static_onnx_temp_dir.
2023-08-15 23:08:31 [INFO] Static PaddlePaddle model saved in model/paddle_model_static_onnx_temp_dir.
.. parsed-literal::
I0711 23:03:56.497538 3455488 interpretercore.cc:267] New Executor is Running.
I0815 23:08:31.864205 2062621 interpretercore.cc:267] New Executor is Running.
.. parsed-literal::
@ -210,18 +239,18 @@ Convert PaddlePaddle Model to ONNX
[Paddle2ONNX] Start to parsing Paddle model...
[Paddle2ONNX] Use opset_version = 13 for ONNX export.
[Paddle2ONNX] PaddlePaddle model is exported as ONNX format now.
2023-07-11 23:04:00 [INFO] ONNX model saved in model/paddlegan_sr.onnx.
2023-08-15 23:08:35 [INFO] ONNX model saved in model/paddlegan_sr.onnx.
Convert ONNX Model to OpenVINO IR with `Model Optimizer Python API <https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_Python_API.html>`__
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Convert ONNX Model to OpenVINO IR with Model Conversion Python API `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
from openvino.tools import mo
from openvino.runtime import serialize
## Uncomment the command below to show Model Optimizer help, which shows the possible arguments for Model Optimizer.
## Uncomment the command below to show help, which shows the possible arguments for model conversion API.
# mo.convert_model(help=True)
.. code:: ipython3
@ -243,17 +272,46 @@ Convert ONNX Model to OpenVINO IR with `Model Optimizer Python API <https://docs
Exporting ONNX model to OpenVINO IR... This may take a few minutes.
Do Inference on OpenVINO IR Model
---------------------------------
Do Inference on OpenVINO IR Model `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
# Read the network and get input and output names.
ie = Core()
core = Core()
# Alternatively, the model obtained from `mo.convert_model()` may be used here
model = ie.read_model(model=ir_path)
model = core.read_model(model=ir_path)
input_layer = model.input(0)
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
.. code:: ipython3
# Load and show the image.
@ -273,18 +331,18 @@ Do Inference on OpenVINO IR Model
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f8eb83c9550>
<matplotlib.image.AxesImage at 0x7f485b8501f0>
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_25_1.png
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_27_1.png
.. code:: ipython3
# Load the network to the CPU device (this may take a few seconds).
compiled_model = ie.compile_model(model=model, device_name="CPU")
compiled_model = core.compile_model(model=model, device_name=device.value)
output_layer = compiled_model.output(0)
.. code:: ipython3
@ -302,7 +360,7 @@ Do Inference on OpenVINO IR Model
.. parsed-literal::
Inference duration: 3.26 seconds
Inference duration: 3.30 seconds
.. code:: ipython3
@ -325,16 +383,17 @@ Do Inference on OpenVINO IR Model
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f8e44040460>
<matplotlib.image.AxesImage at 0x7f47d00b7e50>
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_29_1.png
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_31_1.png
Show an Animated GIF
~~~~~~~~~~~~~~~~~~~~
Show an Animated GIF `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To visualize the difference between the bicubic image and the
superresolution image, create an animated GIF image that switches
@ -362,13 +421,14 @@ between both versions.
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_31_0.png
.. image:: 207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_33_0.png
:width: 960px
Create a Comparison Video
~~~~~~~~~~~~~~~~~~~~~~~~~
Create a Comparison Video `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Create a video with a “slider”, showing the bicubic image to the right
and the superresolution image on the left.

0
docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_25_1.png → docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_27_1.png
View File

0
docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_29_1.png → docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_31_1.png
View File

0
docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_31_0.png → docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/207-vision-paddlegan-superresolution-with-output_33_0.png
View File

10
docs/notebooks/207-vision-paddlegan-superresolution-with-output_files/index.html
View File

@ -1,9 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/207-vision-paddlegan-superresolution-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/207-vision-paddlegan-superresolution-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230711220806/dist/rst_files/207-vision-paddlegan-superresolution-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="207-vision-paddlegan-superresolution-with-output_25_1.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 12-Jul-2023 00:11 436999
<a href="207-vision-paddlegan-superresolution-with-output_29_1.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 12-Jul-2023 00:11 476190
<a href="207-vision-paddlegan-superresolution-with-output_31_0.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 12-Jul-2023 00:11 2835354
<h1>Index of /projects/ov-notebook/0.1.0-latest/20230815220807/dist/rst_files/207-vision-paddlegan-superresolution-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="207-vision-paddlegan-superresolution-with-output_27_1.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 16-Aug-2023 01:31 436999
<a href="207-vision-paddlegan-superresolution-with-output_31_1.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 16-Aug-2023 01:31 476190
<a href="207-vision-paddlegan-superresolution-with-output_33_0.png">207-vision-paddlegan-superresolution-with-outpu..&gt;</a> 16-Aug-2023 01:31 2835354
</pre><hr></body>
</html>

177
docs/notebooks/208-optical-character-recognition-with-output.rst
View File

@ -1,6 +1,8 @@
Optical Character Recognition (OCR) with OpenVINO™
==================================================
.. _top:
This tutorial demonstrates how to perform optical character recognition
(OCR) with OpenVINO models. It is a continuation of the
`004-hello-detection <004-hello-detection-with-output.html>`__
@ -19,8 +21,34 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo>`__. For more
information, refer to the
`104-model-tools <104-model-tools-with-output.html>`__ tutorial.
Imports
-------
**Table of contents**:
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Download Models <#download-models>`__
- `Convert Models <#convert-models>`__
- `Select inference device <#select-inference-device>`__
- `Object Detection <#object-detection>`__
- `Load a Detection Model <#load-a-detection-model>`__
- `Load an Image <#load-an-image>`__
- `Do Inference <#do-inference>`__
- `Get Detection Results <#get-detection-results>`__
- `Text Recognition <#text-recognition>`__
- `Load Text Recognition Model <#load-text-recognition-model>`__
- `Do Inference <#do-inference>`__
- `Show Results <#show-results>`__
- `Show Detected Text Boxes and OCR Results for the Image <#show-detected-text-boxes-and-ocr-results-for-the-image>`__
- `Show the OCR Result per Bounding Box <#show-the-ocr-result-per-bounding-box>`__
- `Print Annotations in Plain Text Format <#print-annotations-in-plain-text-format>`__
Imports `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
@ -37,12 +65,13 @@ Imports
sys.path.append("../utils")
from notebook_utils import load_image
Settings
--------
Settings `⇑ <#top>`__
###############################################################################################################################
.. code:: ipython3
ie = Core()
core = Core()
model_dir = Path("model")
precision = "FP16"
@ -51,8 +80,9 @@ Settings
model_dir.mkdir(exist_ok=True)
Download Models
---------------
Download Models `⇑ <#top>`__
###############################################################################################################################
The next cells will run Model Downloader to download the detection and
recognition models. If the models have been downloaded before, they will
@ -250,6 +280,8 @@ Downloading horizontal-text-detection-0001, text-recognition-resnet-fc…
========== Replacing text in model/public/text-recognition-resnet-fc/vedastr/utils/config.py
========== Replacing text in model/public/text-recognition-resnet-fc/vedastr/utils/config.py
========== Replacing text in model/public/text-recognition-resnet-fc/vedastr/utils/config.py
========== Replacing text in model/public/text-recognition-resnet-fc/vedastr/models/bodies/feature_extractors/encoders/backbones/resnet.py
========== Replacing text in model/public/text-recognition-resnet-fc/vedastr/models/bodies/feature_extractors/encoders/backbones/resnet.py
========== Unpacking model/public/text-recognition-resnet-fc/vedastr/addict-2.4.0-py3-none-any.whl
@ -267,8 +299,9 @@ text-recognition-resnet-fc.
# for line in download_result:
# print(line)
Convert Models
--------------
Convert Models `⇑ <#top>`__
###############################################################################################################################
The downloaded detection model is an Intel model, which is already in
OpenVINO Intermediate Representation (OpenVINO IR) format. The text
@ -300,45 +333,74 @@ Converting text-recognition-resnet-fc…
.. parsed-literal::
========== Converting text-recognition-resnet-fc to ONNX
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-path=/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/text-recognition-resnet-fc --model-path=model/public/text-recognition-resnet-fc --model-name=get_model --import-module=model '--model-param=file_config=r"model/public/text-recognition-resnet-fc/vedastr/configs/resnet_fc.py"' '--model-param=weights=r"model/public/text-recognition-resnet-fc/vedastr/ckpt/resnet_fc.pth"' --input-shape=1,1,32,100 --input-names=input --output-names=output --output-file=model/public/text-recognition-resnet-fc/resnet_fc.onnx
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-path=/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/text-recognition-resnet-fc --model-path=model/public/text-recognition-resnet-fc --model-name=get_model --import-module=model '--model-param=file_config=r"model/public/text-recognition-resnet-fc/vedastr/configs/resnet_fc.py"' '--model-param=weights=r"model/public/text-recognition-resnet-fc/vedastr/ckpt/resnet_fc.pth"' --input-shape=1,1,32,100 --input-names=input --output-names=output --output-file=model/public/text-recognition-resnet-fc/resnet_fc.onnx
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchvision/models/_utils.py:252: UserWarning: Accessing the model URLs via the internal dictionary of the module is deprecated since 0.13 and may be removed in the future. Please access them via the appropriate Weights Enum instead.
warnings.warn(
ONNX check passed successfully.
========== Converting text-recognition-resnet-fc to IR (FP16)
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-448/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=/tmp/tmpvegltj_7 --model_name=text-recognition-resnet-fc --input=input '--mean_values=input[127.5]' '--scale_values=input[127.5]' --output=output --input_model=model/public/text-recognition-resnet-fc/resnet_fc.onnx '--layout=input(NCHW)' '--input_shape=[1, 1, 32, 100]' --compress_to_fp16=True
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-475/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=/tmp/tmppkwl27u7 --model_name=text-recognition-resnet-fc --input=input '--mean_values=input[127.5]' '--scale_values=input[127.5]' --output=output --input_model=model/public/text-recognition-resnet-fc/resnet_fc.onnx '--layout=input(NCHW)' '--input_shape=[1, 1, 32, 100]' --compress_to_fp16=True
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument --compress_to_fp16 or set it to false --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/latest/openvino_docs_MO_DG_FP16_Compression.html
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/latest/openvino_2_0_transition_guide.html
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /tmp/tmpvegltj_7/text-recognition-resnet-fc.xml
[ SUCCESS ] BIN file: /tmp/tmpvegltj_7/text-recognition-resnet-fc.bin
[ SUCCESS ] XML file: /tmp/tmppkwl27u7/text-recognition-resnet-fc.xml
[ SUCCESS ] BIN file: /tmp/tmppkwl27u7/text-recognition-resnet-fc.bin
Object Detection
----------------
Select inference device `⇑ <#top>`__
###############################################################################################################################
Select device from dropdown list for running inference using OpenVINO:
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
Object Detection `⇑ <#top>`__
###############################################################################################################################
Load a detection model, load an image, do inference and get the
detection inference result.
Load a Detection Model
~~~~~~~~~~~~~~~~~~~~~~
Load a Detection Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
detection_model = ie.read_model(
detection_model = core.read_model(
model=detection_model_path, weights=detection_model_path.with_suffix(".bin")
)
detection_compiled_model = ie.compile_model(model=detection_model, device_name="CPU")
detection_compiled_model = core.compile_model(model=detection_model, device_name=device.value)
detection_input_layer = detection_compiled_model.input(0)
Load an Image
~~~~~~~~~~~~~
Load an Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -360,11 +422,12 @@ Load an Image
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_13_0.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_15_0.png
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Text boxes are detected in the images and returned as blobs of data in
the shape of ``[100, 5]``. Each description of detection has the
@ -378,8 +441,9 @@ the shape of ``[100, 5]``. Each description of detection has the
# Remove zero only boxes.
boxes = boxes[~np.all(boxes == 0, axis=1)]
Get Detection Results
~~~~~~~~~~~~~~~~~~~~~
Get Detection Results `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -447,22 +511,24 @@ Get Detection Results
return rgb_image
Text Recogntion
---------------
Text Recognition `⇑ <#top>`__
###############################################################################################################################
Load the text recognition model and do inference on the detected boxes
from the detection model.
Load Text Recognition Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load Text Recognition Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
recognition_model = ie.read_model(
recognition_model = core.read_model(
model=recognition_model_path, weights=recognition_model_path.with_suffix(".bin")
)
recognition_compiled_model = ie.compile_model(model=recognition_model, device_name="CPU")
recognition_compiled_model = core.compile_model(model=recognition_model, device_name=device.value)
recognition_output_layer = recognition_compiled_model.output(0)
recognition_input_layer = recognition_compiled_model.input(0)
@ -470,8 +536,9 @@ Load Text Recognition Model
# Get the height and width of the input layer.
_, _, H, W = recognition_input_layer.shape
Do Inference
~~~~~~~~~~~~
Do Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
@ -516,11 +583,13 @@ Do Inference
boxes_with_annotations = list(zip(boxes, annotations))
Show Results
------------
Show Results `⇑ <#top>`__
###############################################################################################################################
Show Detected Text Boxes and OCR Results for the Image `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Show Detected Text Boxes and OCR Results for the Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Visualize the result by drawing boxes around recognized text and showing
the OCR result from the text recognition model.
@ -532,11 +601,12 @@ the OCR result from the text recognition model.
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_23_0.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_0.png
Show the OCR Result per Bounding Box
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Show the OCR Result per Bounding Box `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Depending on the image, the OCR result may not be readable in the image
with boxes, as displayed in the cell above. Use the code below to
@ -549,7 +619,7 @@ display the extracted boxes and the OCR result per box.
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_0.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_0.png
@ -557,7 +627,7 @@ building
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_2.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_2.png
@ -565,7 +635,7 @@ noyce
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_4.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_4.png
@ -573,7 +643,7 @@ noyce
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_6.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_6.png
@ -581,7 +651,7 @@ n
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_8.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_8.png
@ -589,15 +659,16 @@ center
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_10.png
.. image:: 208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_10.png
robert
Print Annotations in Plain Text Format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Print Annotations in Plain Text Format `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Print annotations for detected text based on their position in the input
image, starting from the upper left corner.

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_13_0.png → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_15_0.png
View File

3
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_23_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3cf5c39f43e8ae8b58bb2cf83ebb5d94043a715d59e1f1a5641a83742d17b7a4
size 923631

4
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3afa4268972711c6dd52892c4fa0f9600df3ae40bbec42d5867b83fd75f9308c
size 11367
oid sha256:3cf5c39f43e8ae8b58bb2cf83ebb5d94043a715d59e1f1a5641a83742d17b7a4
size 923631

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_0.jpg → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_0.jpg
View File

3
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_0.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3afa4268972711c6dd52892c4fa0f9600df3ae40bbec42d5867b83fd75f9308c
size 11367

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_10.jpg → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_10.jpg
View File

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_10.png → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_10.png
View File

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_2.jpg → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_2.jpg
View File

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_2.png → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_2.png
View File

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_4.jpg → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_4.jpg
View File

0
docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_25_4.png → docs/notebooks/208-optical-character-recognition-with-output_files/208-optical-character-recognition-with-output_27_4.png
View File

Some files were not shown because too many files have changed in this diff Show More