natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] Updating interactive tutorials (#22494) 

* Updating Interactive Tutorials

* Updating Tutorials
This commit is contained in:
Sebastian Golebiewski 2024-01-29 10:49:32 +01:00 committed by GitHub
parent d48ee3553b
commit 45a9992aee
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
512 changed files with 102296 additions and 12196 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

74
docs/articles_en/learn_openvino/tutorials.rst
View File

@ -59,6 +59,45 @@ The Jupyter notebooks are categorized into following classes:
Below you will find a selection of recommended tutorials that demonstrate inference on a particular model. These tutorials are guaranteed to provide a great experience with inference in OpenVINO:
.. showcase::
:title: 280-depth-anything
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/280-depth-anything/280-depth-anything.gif
Depth estimation with DepthAnything and OpenVINO.
.. showcase::
:title: 279-mobilevlm-language-assistant
:img: _static/images/notebook_eye.png
Mobile language assistant with MobileVLM and OpenVINO.
.. showcase::
:title: 278-stable-diffusion-ip-adapter
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/278-stable-diffusion-ip-adapter/278-stable-diffusion-ip-adapter.png
Image Generation with Stable Diffusion and IP-Adapter.
.. showcase::
:title: 275-llm-question-answering
:img: _static/images/notebook_eye.png
LLM Instruction-following pipeline with OpenVINO.
.. showcase::
:title: 274-efficient-sam
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/274-efficient-sam/274-efficient-sam.png
Object segmentations with EfficientSAM and OpenVINO.
.. showcase::
:title: 273-stable-zephyr-3b-chatbot
:img: _static/images/notebook_eye.png
LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO.
.. showcase::
:title: 272-paint-by-example
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/272-paint-by-example/272-paint-by-example.png
@ -83,41 +122,6 @@ Below you will find a selection of recommended tutorials that demonstrate infere
Frame interpolation using FILM and OpenVINO.
.. showcase::
:title: 267-distil-whisper-asr
:img: _static/images/notebook_eye.png
Automatic speech recognition using Distil-Whisper and OpenVINO.
.. showcase::
:title: 265-wuerstchen-image-generation
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/265-wuerstchen-image-generation/265-wuerstchen-image-generation.png
Image generation with Würstchen and OpenVINO.
.. showcase::
:title: 264-qrcode-monster
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/264-qrcode-monster/264-qrcode-monster.png
Generate creative QR codes with ControlNet QR Code Monster and OpenVINO.
.. showcase::
:title: 263-latent-consistency-models-image-generation
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/263-latent-consistency-models-image-generation/263-latent-consistency-models-image-generation.png
Image generation with Latent Consistency Model and OpenVINO.
.. showcase::
:title: 263-lcm-lora-controlnet
:img: https://user-images.githubusercontent.com/29454499/284292122-f146e16d-7233-49f7-a401-edcb714b5288.png
Text-to-Image Generation with LCM LoRA and ControlNet Conditioning.
.. showcase::
:title: 262-softvc-voice-conversion
:img: _static/images/notebook_eye.png
SoftVC VITS Singing Voice Conversion and OpenVINO.
.. note::

8
docs/articles_en/learn_openvino/tutorials/notebooks_section_1.rst
View File

@ -53,11 +53,17 @@ Tutorials that explain how to optimize and quantize models with OpenVINO tools.
Quantize Speech Recognition Models with accuracy control using NNCF PTQ API.
.. showcase::
:title: 121-legacy-mo-convert-to-openvino
:img: _static/images/notebook_eye.png
Learn about OpenVINO™ model conversion API.
.. showcase::
:title: 121-convert-to-openvino
:img: _static/images/notebook_eye.png
Learn OpenVINO model conversion API.
Learn about model conversion in OpenVINO™.
.. showcase::
:title: 120-tensorflow-object-detection-to-openvino

68
docs/articles_en/learn_openvino/tutorials/notebooks_section_2.rst
View File

@ -11,6 +11,54 @@ Model Demos
Demos that demonstrate inference on a particular model.
.. showcase::
:title: 280-depth-anything
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/280-depth-anything/280-depth-anything.gif
Depth estimation with DepthAnything and OpenVINO.
.. showcase::
:title: 279-mobilevlm-language-assistant
:img: _static/images/notebook_eye.png
Mobile language assistant with MobileVLM and OpenVINO.
.. showcase::
:title: 278-stable-diffusion-ip-adapter
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/278-stable-diffusion-ip-adapter/278-stable-diffusion-ip-adapter.png
Image Generation with Stable Diffusion and IP-Adapter.
.. showcase::
:title: 277-amused-lightweight-text-to-image
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/277-amused-lightweight-text-to-image/277-amused-lightweight-text-to-image.png
Lightweight image generation with aMUSEd and OpenVINO.
.. showcase::
:title: 276-stable-diffusion-torchdynamo-backend
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/276-stable-diffusion-torchdynamo-backend/276-stable-diffusion-torchdynamo-backend.png
Image Generation with Stable Diffusion using OpenVINO TorchDynamo backend.
.. showcase::
:title: 275-llm-question-answering
:img: _static/images/notebook_eye.png
LLM Instruction-following pipeline with OpenVINO.
.. showcase::
:title: 274-efficient-sam
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/274-efficient-sam/274-efficient-sam.png
Object segmentations with EfficientSAM and OpenVINO.
.. showcase::
:title: 273-stable-zephyr-3b-chatbot
:img: _static/images/notebook_eye.png
LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO.
.. showcase::
:title: 272-paint-by-example
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/272-paint-by-example/272-paint-by-example.png
@ -105,12 +153,24 @@ Demos that demonstrate inference on a particular model.
Visual-language assistant with LLaVA and OpenVINO.
.. showcase::
:title: 257-videollava-multimodal-chatbot.ipynb
:img: _static/images/notebook_eye.png
Visual-language assistant with Video-LLaVA and OpenVINO.
.. showcase::
:title: 256-bark-text-to-audio
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/256-bark-text-to-audio/256-bark-text-to-audio.png
Text-to-speech generation using Bark and OpenVINO.
.. showcase::
:title: 254-rag-chatbot
:img: _static/images/notebook_eye.png
Create an LLM-powered RAG system using OpenVINO.
.. showcase::
:title: 254-llm-chatbot
:img: _static/images/notebook_eye.png
@ -147,9 +207,15 @@ Demos that demonstrate inference on a particular model.
Universal segmentation with OneFormer and OpenVINO™.
.. showcase::
:title: 248-segmind-vegart
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.png
High-resolution image generation with Segmind-VegaRT and OpenVINO.
.. showcase::
:title: 248-ssd-b1
:img: https://user-images.githubusercontent.com/29454499/258651862-28b63016-c5ff-4263-9da8-73ca31100165.jpeg
:img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.png
Image generation with Stable Diffusion XL and OpenVINO™.

2
docs/nbdoc/consts.py
View File

@ -5,7 +5,7 @@ repo_directory = "notebooks"
repo_owner = "openvinotoolkit"
repo_name = "openvino_notebooks"
repo_branch = "tree/main"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/"
blacklisted_extensions = ['.xml', '.bin']
notebooks_repo = "https://github.com/openvinotoolkit/openvino_notebooks/blob/main/"
notebooks_binder = "https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath="

45
docs/notebooks/001-hello-world-with-output.rst
View File

@ -5,7 +5,7 @@ This basic introduction to OpenVINO™ shows how to do inference with an
image classification model.
A pre-trained `MobileNetV3
model <https://docs.openvino.ai/2023.3/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
model <https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
from `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ is used in
this tutorial. For more information about how OpenVINO IR models are
@ -13,11 +13,12 @@ created, refer to the `TensorFlow to
OpenVINO <101-tensorflow-classification-to-openvino-with-output.html>`__
tutorial.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Download the Model and data samples <#download-the-model-and-data-samples>`__
- `Download the Model and data
samples <#download-the-model-and-data-samples>`__
- `Select inference device <#select-inference-device>`__
- `Load the Model <#load-the-model>`__
- `Load an Image <#load-an-image>`__
@ -34,8 +35,10 @@ tutorial.
Note: you may need to restart the kernel to use updated packages.
Imports
-------------------------------------------------
Imports
-------
.. code:: ipython3
@ -55,8 +58,10 @@ Imports
from notebook_utils import download_file
Download the Model and data samples
-----------------------------------------------------------------------------
Download the Model and data samples
-----------------------------------
.. code:: ipython3
@ -89,8 +94,10 @@ Download the Model and data samples
artifacts/v3-small_224_1.0_float.bin: 0%| | 0.00/4.84M [00:00<?, ?B/s]
Select inference device
-----------------------------------------------------------------
Select inference device
-----------------------
select device from dropdown list for running inference using OpenVINO
@ -117,8 +124,10 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------------------------------------------------
Load the Model
--------------
.. code:: ipython3
@ -128,8 +137,10 @@ Load the Model
output_layer = compiled_model.output(0)
Load an Image
-------------------------------------------------------
Load an Image
-------------
.. code:: ipython3
@ -160,8 +171,10 @@ Load an Image
.. image:: 001-hello-world-with-output_files/001-hello-world-with-output_11_1.png
Do Inference
------------------------------------------------------
Do Inference
------------
.. code:: ipython3

2
docs/notebooks/001-hello-world-with-output_files/001-hello-world-with-output_11_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7511b8a4e5b047600d5fed14fbc7e9653a868bc5253abf1e0c3ef649b47bc408
oid sha256:2dd4338c6c163e7693885ce544e8c9cd2aecedf3b136fa295e22877f37b5634c
size 387941

7
docs/notebooks/001-hello-world-with-output_files/index.html
View File

@ -1,7 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/001-hello-world-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/001-hello-world-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="001-hello-world-with-output_11_1.png">001-hello-world-with-output_11_1.png</a> 31-Oct-2023 00:35 387941
</pre><hr></body>
</html>

7
docs/notebooks/001-hello-world-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/001-hello-world-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/001-hello-world-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="001-hello-world-with-output_11_1.png">001-hello-world-with-output_11_1.png</a> 26-Jan-2024 01:05 387941
</pre><hr></body>
</html>

120
docs/notebooks/002-openvino-api-with-output.rst
View File

@ -10,8 +10,8 @@ used in this tutorial are provided as examples. These model files can be
replaced with your own models. The exact outputs will be different, but
the process is the same.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Loading OpenVINO Runtime and Showing
Info <#loading-openvino-runtime-and-showing-info>`__
@ -57,35 +57,55 @@ the process is the same.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.1)
Requirement already satisfied: ipywidgets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (8.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.11.17)
Requirement already satisfied: comm>=0.1.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (0.2.0)
Requirement already satisfied: ipython>=6.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (8.12.3)
Requirement already satisfied: traitlets>=4.3.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (5.14.0)
Requirement already satisfied: widgetsnbextension~=4.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (4.0.9)
Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (3.0.9)
Requirement already satisfied: backcall in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: decorator in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1)
Requirement already satisfied: matplotlib-inline in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)
Requirement already satisfied: pickleshare in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.41)
Requirement already satisfied: pygments>=2.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.17.2)
Requirement already satisfied: stack-data in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.3)
Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)
Requirement already satisfied: pexpect>4.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.12)
Requirement already satisfied: executing>=1.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1)
Requirement already satisfied: pure-eval in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2)
Requirement already satisfied: six>=1.12.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0)
.. parsed-literal::
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.1)
Requirement already satisfied: ipywidgets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (8.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.11.17)
.. parsed-literal::
Requirement already satisfied: comm>=0.1.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (0.2.1)
Requirement already satisfied: ipython>=6.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (8.12.3)
Requirement already satisfied: traitlets>=4.3.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (5.14.1)
Requirement already satisfied: widgetsnbextension~=4.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (4.0.9)
Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (3.0.9)
Requirement already satisfied: backcall in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: decorator in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1)
Requirement already satisfied: matplotlib-inline in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)
Requirement already satisfied: pickleshare in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.43)
Requirement already satisfied: pygments>=2.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.17.2)
Requirement already satisfied: stack-data in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.3)
Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)
.. parsed-literal::
Requirement already satisfied: pexpect>4.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.13)
Requirement already satisfied: executing>=1.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1)
Requirement already satisfied: pure-eval in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2)
.. parsed-literal::
Requirement already satisfied: six>=1.12.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0)
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -137,7 +157,7 @@ After initializing OpenVINO Runtime, first read the model file with
``compile_model()`` method.
`OpenVINO™ supports several model
formats <https://docs.openvino.ai/2023.3/Supported_Model_Formats_MO_DG.html#doxid-supported-model-formats>`__
formats <https://docs.openvino.ai/2023.3/Supported_Model_Formats.html>`__
and enables developers to convert them to its own OpenVINO IR format
using a tool dedicated to this task.
@ -198,7 +218,7 @@ notebooks.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -247,7 +267,7 @@ points to the filename of an ONNX model.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx')
@ -303,7 +323,7 @@ without any conversion step. Pass the filename with extension to
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams')
@ -347,7 +367,7 @@ TensorFlow models saved in frozen graph format can also be passed to
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb')
@ -399,7 +419,7 @@ It is pre-trained model optimized to work with TensorFlow Lite.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite')
@ -424,7 +444,7 @@ PyTorch Model
`PyTorch <https://pytorch.org/>`__ models can not be directly passed to
``core.read_model``. ``ov.Model`` for model objects from this framework
can be obtained using ``ov.convert_model`` API. You can find more
details in `pytorch-to-openvino <../102-pytorch-to-openvino>`__
details in `pytorch-to-openvino <102-pytorch-to-openvino-with-output.html>`__
notebook. In this tutorial we will use
`resnet18 <https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html>`__
model form torchvision library. After conversion model using
@ -473,6 +493,10 @@ Information about the inputs and outputs of the model are in
.. parsed-literal::
'model/classification.xml' already exists.
.. parsed-literal::
'model/classification.bin' already exists.
@ -480,7 +504,7 @@ Information about the inputs and outputs of the model are in
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -679,6 +703,10 @@ produced data as values.
.. parsed-literal::
'model/classification.xml' already exists.
.. parsed-literal::
'model/classification.bin' already exists.
@ -686,7 +714,7 @@ produced data as values.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
@ -875,7 +903,7 @@ input shape.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin')
@ -911,6 +939,10 @@ input shape.
~~~~ ORIGINAL MODEL ~~~~
input shape: [1,3,512,512]
output shape: [1,1,512,512]
.. parsed-literal::
~~~~ RESHAPED MODEL ~~~~
model input shape: [1,3,544,544]
compiled_model input shape: [1,3,544,544]
@ -1023,6 +1055,10 @@ the cache.
.. parsed-literal::
'model/classification.xml' already exists.
.. parsed-literal::
'model/classification.bin' already exists.
@ -1030,7 +1066,7 @@ the cache.
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin')

110
docs/notebooks/003-hello-segmentation-with-output.rst
View File

@ -4,14 +4,14 @@ Hello Image Segmentation
A very basic introduction to using segmentation models with OpenVINO™.
In this tutorial, a pre-trained
`road-segmentation-adas-0001 <https://docs.openvino.ai/2023.3/omz_models_model_road_segmentation_adas_0001.html>`__
`road-segmentation-adas-0001 <https://docs.openvino.ai/2023.0/omz_models_model_road_segmentation_adas_0001.html>`__
model from the `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ is used.
ADAS stands for Advanced Driver Assistance Services. The model
recognizes four classes: background, road, curb and mark.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Download model weights <#download-model-weights>`__
@ -33,8 +33,10 @@ recognizes four classes: background, road, curb and mark.
Note: you may need to restart the kernel to use updated packages.
Imports
-------------------------------------------------
Imports
-------
.. code:: ipython3
@ -42,35 +44,37 @@ Imports
import matplotlib.pyplot as plt
import numpy as np
import openvino as ov
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py',
filename='notebook_utils.py'
)
from notebook_utils import segmentation_map_to_image, download_file
Download model weights
----------------------------------------------------------------
Download model weights
----------------------
.. code:: ipython3
from pathlib import Path
base_model_dir = Path("./model").expanduser()
model_name = "road-segmentation-adas-0001"
model_xml_name = f'{model_name}.xml'
model_bin_name = f'{model_name}.bin'
model_xml_path = base_model_dir / model_xml_name
if not model_xml_path.exists():
model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.xml"
model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.bin"
download_file(model_xml_url, model_xml_name, base_model_dir)
download_file(model_bin_url, model_bin_name, base_model_dir)
else:
@ -89,15 +93,17 @@ Download model weights
model/road-segmentation-adas-0001.bin: 0%| | 0.00/720k [00:00<?, ?B/s]
Select inference device
-----------------------------------------------------------------
Select inference device
-----------------------
select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
@ -105,7 +111,7 @@ select device from dropdown list for running inference using OpenVINO
description='Device:',
disabled=False,
)
device
@ -117,25 +123,27 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------------------------------------------------
Load the Model
--------------
.. code:: ipython3
core = ov.Core()
model = core.read_model(model=model_xml_path)
compiled_model = core.compile_model(model=model, device_name=device.value)
input_layer_ir = compiled_model.input(0)
output_layer_ir = compiled_model.output(0)
Load an Image
-------------------------------------------------------
Load an Image
-------------
A sample image from the `Mapillary
Vistas <https://www.mapillary.com/dataset/vistas>`__ dataset is
provided.
A sample image from the
`Mapillary Vistas <https://www.mapillary.com/dataset/vistas>`__ dataset
is provided.
.. code:: ipython3
@ -144,23 +152,23 @@ provided.
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/empty_road_mapillary.jpg",
directory="data"
)
# The segmentation network expects images in BGR format.
image = cv2.imread(str(image_filename))
rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image_h, image_w, _ = image.shape
# N,C,H,W = batch size, number of channels, height, width.
N, C, H, W = input_layer_ir.shape
# OpenCV resize expects the destination size as (width, height).
resized_image = cv2.resize(image, (W, H))
# Reshape to the network input shape.
input_image = np.expand_dims(
resized_image.transpose(2, 0, 1), 0
)
)
plt.imshow(rgb_image)
@ -174,7 +182,7 @@ provided.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f42f835ec10>
<matplotlib.image.AxesImage at 0x7ffab9d92970>
@ -182,14 +190,16 @@ provided.
.. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png
Do Inference
------------------------------------------------------
Do Inference
------------
.. code:: ipython3
# Run the inference.
result = compiled_model([input_image])[output_layer_ir]
# Prepare data for visualization.
segmentation_mask = np.argmax(result, axis=1)
plt.imshow(segmentation_mask.transpose(1, 2, 0))
@ -199,7 +209,7 @@ Do Inference
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f42f823fa90>
<matplotlib.image.AxesImage at 0x7ffa747040d0>
@ -207,41 +217,45 @@ Do Inference
.. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png
Prepare Data for Visualization
------------------------------------------------------------------------
Prepare Data for Visualization
------------------------------
.. code:: ipython3
# Define colormap, each color represents a class.
colormap = np.array([[68, 1, 84], [48, 103, 141], [53, 183, 120], [199, 216, 52]])
# Define the transparency of the segmentation mask on the photo.
alpha = 0.3
# Use function from notebook_utils.py to transform mask to an RGB image.
mask = segmentation_map_to_image(segmentation_mask, colormap)
resized_mask = cv2.resize(mask, (image_w, image_h))
# Create an image with mask.
image_with_mask = cv2.addWeighted(resized_mask, alpha, rgb_image, 1 - alpha, 0)
Visualize data
--------------------------------------------------------
Visualize data
--------------
.. code:: ipython3
# Define titles with images.
data = {"Base Photo": rgb_image, "Segmentation": mask, "Masked Photo": image_with_mask}
# Create a subplot to visualize images.
fig, axs = plt.subplots(1, len(data.items()), figsize=(15, 10))
# Fill the subplot.
for ax, (name, image) in zip(axs, data.items()):
ax.axis('off')
ax.set_title(name)
ax.imshow(image)
# Display an image.
plt.show(fig)

2
docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:76113c575caa9c8a8aca45d3ec6ebd7a4b513dadffd8e9e63861a7a041d7e5de
oid sha256:90fac1140a8f375892b9637d611b5b4d7e51f78a45f3af3ab3b33e74b97baa18
size 249032

2
docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d9d40b840eea67857e42a78dd89b65799215e806e4f8d23b0523b0c163cb453b
oid sha256:57f846855a237159135e3491021459d4d94b97ddb0458c74b7bc35cca5ff0d5a
size 20550

2
docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_17_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6652ac31dc4e31bcb618c0c13795dac83fa49291a436aea8058847f179b4095d
oid sha256:655bbfae2c0894ca52bb1cdaf7de64bbad747265984e968afc86f0198bc6600d
size 260045

10
docs/notebooks/003-hello-segmentation-with-output_files/index.html → docs/notebooks/003-hello-segmentation-with-output_files/index.html.tmp
View File

@ -1,9 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/003-hello-segmentation-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/003-hello-segmentation-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/003-hello-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="003-hello-segmentation-with-output_11_2.png">003-hello-segmentation-with-output_11_2.png</a> 31-Oct-2023 00:35 249032
<a href="003-hello-segmentation-with-output_13_1.png">003-hello-segmentation-with-output_13_1.png</a> 31-Oct-2023 00:35 20550
<a href="003-hello-segmentation-with-output_17_0.png">003-hello-segmentation-with-output_17_0.png</a> 31-Oct-2023 00:35 260045
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/003-hello-segmentation-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="003-hello-segmentation-with-output_11_2.png">003-hello-segmentation-with-output_11_2.png</a> 26-Jan-2024 01:05 249032
<a href="003-hello-segmentation-with-output_13_1.png">003-hello-segmentation-with-output_13_1.png</a> 26-Jan-2024 01:05 20550
<a href="003-hello-segmentation-with-output_17_0.png">003-hello-segmentation-with-output_17_0.png</a> 26-Jan-2024 01:05 260045
</pre><hr></body>
</html>

48
docs/notebooks/004-hello-detection-with-output.rst
View File

@ -5,7 +5,7 @@ A very basic introduction to using object detection models with
OpenVINO™.
The
`horizontal-text-detection-0001 <https://docs.openvino.ai/2023.3/omz_models_model_horizontal_text_detection_0001.html>`__
`horizontal-text-detection-0001 <https://docs.openvino.ai/2023.0/omz_models_model_horizontal_text_detection_0001.html>`__
model from `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__ is used. It
detects horizontal text in images and returns a blob of data in the
@ -16,8 +16,8 @@ corner, ``(x_max, y_max)`` are the coordinates of the bottom right
bounding box corner and ``conf`` is the confidence for the predicted
class.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Download model weights <#download-model-weights>`__
@ -38,8 +38,10 @@ class.
Note: you may need to restart the kernel to use updated packages.
Imports
-------------------------------------------------
Imports
-------
.. code:: ipython3
@ -58,8 +60,10 @@ Imports
from notebook_utils import download_file
Download model weights
----------------------------------------------------------------
Download model weights
----------------------
.. code:: ipython3
@ -94,8 +98,10 @@ Download model weights
model/horizontal-text-detection-0001.bin: 0%| | 0.00/7.39M [00:00<?, ?B/s]
Select inference device
-----------------------------------------------------------------
Select inference device
-----------------------
select device from dropdown list for running inference using OpenVINO
@ -122,8 +128,10 @@ select device from dropdown list for running inference using OpenVINO
Load the Model
--------------------------------------------------------
Load the Model
--------------
.. code:: ipython3
@ -135,8 +143,10 @@ Load the Model
input_layer_ir = compiled_model.input(0)
output_layer_ir = compiled_model.output("boxes")
Load an Image
-------------------------------------------------------
Load an Image
-------------
.. code:: ipython3
@ -171,8 +181,10 @@ Load an Image
.. image:: 004-hello-detection-with-output_files/004-hello-detection-with-output_11_1.png
Do Inference
------------------------------------------------------
Do Inference
------------
.. code:: ipython3
@ -182,8 +194,10 @@ Do Inference
# Remove zero only boxes.
boxes = boxes[~np.all(boxes == 0, axis=1)]
Visualize Results
-----------------------------------------------------------
Visualize Results
-----------------
.. code:: ipython3

2
docs/notebooks/004-hello-detection-with-output_files/004-hello-detection-with-output_11_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69513616afb235a4e034b385ef3d3c9ab047e73c0ea58bd8c055e7da01b3bda0
oid sha256:538f477cad7f1fad28669553d8cac16e28f403c0eab9e95a2546b071dd2fbfa1
size 305482

2
docs/notebooks/004-hello-detection-with-output_files/004-hello-detection-with-output_16_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:05843bab5f2457b274c4178bfa173fc55b6403bbae29d10d4a7e91fddc818bd5
oid sha256:18a797607c8b3a7b14e6949e480c9ccd6a9629fa2026cdedfe5c4b6ab891edbd
size 457214

8
docs/notebooks/004-hello-detection-with-output_files/index.html
View File

@ -1,8 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/004-hello-detection-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/004-hello-detection-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="004-hello-detection-with-output_11_1.png">004-hello-detection-with-output_11_1.png</a> 31-Oct-2023 00:35 305482
<a href="004-hello-detection-with-output_16_0.png">004-hello-detection-with-output_16_0.png</a> 31-Oct-2023 00:35 457214
</pre><hr></body>
</html>

8
docs/notebooks/004-hello-detection-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/004-hello-detection-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/004-hello-detection-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="004-hello-detection-with-output_11_1.png">004-hello-detection-with-output_11_1.png</a> 26-Jan-2024 01:05 305482
<a href="004-hello-detection-with-output_16_0.png">004-hello-detection-with-output_16_0.png</a> 26-Jan-2024 01:05 457214
</pre><hr></body>
</html>

39
docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst
View File

@ -2,7 +2,7 @@ Convert a TensorFlow Model to OpenVINO™
=======================================
This short tutorial shows how to convert a TensorFlow
`MobileNetV3 <https://docs.openvino.ai/2023.3/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
`MobileNetV3 <https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html>`__
image classification model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using `Model Conversion
@ -11,7 +11,8 @@ After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Settings <#settings>`__
@ -74,10 +75,14 @@ Imports
.. parsed-literal::
2023-11-14 22:30:46.626761: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-14 22:30:46.661288: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 22:33:56.723840: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 22:33:56.757735: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-14 22:30:47.171314: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 22:33:57.273134: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Settings
@ -117,12 +122,12 @@ and save it to the disk.
.. parsed-literal::
2023-11-14 22:30:50.201471: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2023-11-14 22:30:50.201504: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2023-11-14 22:30:50.201508: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2023-11-14 22:30:50.201646: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2023-11-14 22:30:50.201662: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2023-11-14 22:30:50.201665: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
2024-01-25 22:34:00.136277: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2024-01-25 22:34:00.136313: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2024-01-25 22:34:00.136317: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2024-01-25 22:34:00.136451: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2024-01-25 22:34:00.136466: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2024-01-25 22:34:00.136470: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
.. parsed-literal::
@ -132,9 +137,13 @@ and save it to the disk.
.. parsed-literal::
2023-11-14 22:30:54.370304: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
2024-01-25 22:34:04.279915: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
[[{{node inputs}}]]
2023-11-14 22:30:57.509389: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
.. parsed-literal::
2024-01-25 22:34:07.400979: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024]
[[{{node inputs}}]]
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 54). These functions will not be directly callable after loading.
@ -165,7 +174,7 @@ model directory and returns OpenVINO Model class instance which
represents this model. Obtained model is ready to use and to be loaded
on a device using ``ov.compile_model`` or can be saved on a disk using
the ``ov.save_model`` function. See the
`tutorial <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__
`tutorial <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__
for more information about using model conversion API with TensorFlow
models.
@ -353,5 +362,5 @@ performance.
.. parsed-literal::
IR model in OpenVINO Runtime/CPU: 0.0010 seconds per image, FPS: 962.52
IR model in OpenVINO Runtime/CPU: 0.0011 seconds per image, FPS: 933.95

2
docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_19_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7511b8a4e5b047600d5fed14fbc7e9653a868bc5253abf1e0c3ef649b47bc408
oid sha256:2dd4338c6c163e7693885ce544e8c9cd2aecedf3b136fa295e22877f37b5634c
size 387941

6
docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html → docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html.tmp
View File

@ -1,7 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="101-tensorflow-classification-to-openvino-with-output_19_1.png">101-tensorflow-classification-to-openvino-with-..&gt;</a> 15-Nov-2023 00:43 387941
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="101-tensorflow-classification-to-openvino-with-output_19_1.png">101-tensorflow-classification-to-openvino-with-..&gt;</a> 26-Jan-2024 01:05 387941
</pre><hr></body>
</html>

24
docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst
View File

@ -33,8 +33,8 @@ plant, sheep, sofa, train, tv monitor**
More information about the model is available in the `torchvision
documentation <https://pytorch.org/vision/main/models/lraspp.html>`__
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Preparation <#preparation>`__
@ -56,10 +56,10 @@ documentation <https://pytorch.org/vision/main/models/lraspp.html>`__
model <#load-the-openvino-ir-network-and-run-inference-on-the-onnx-model>`__
- `1. ONNX Model in OpenVINO
Runtime <#-onnx-model-in-openvino-runtime>`__
Runtime <#1--onnx-model-in-openvino-runtime>`__
- `Select inference device <#select-inference-device>`__
- `2. OpenVINO IR Model in OpenVINO
Runtime <#-openvino-ir-model-in-openvino-runtime>`__
Runtime <#2--openvino-ir-model-in-openvino-runtime>`__
- `Select inference device <#select-inference-device>`__
- `PyTorch Comparison <#pytorch-comparison>`__
@ -576,9 +576,17 @@ performance.
.. parsed-literal::
PyTorch model on CPU: 0.039 seconds per image, FPS: 25.93
ONNX model in OpenVINO Runtime/CPU: 0.018 seconds per image, FPS: 56.39
OpenVINO IR model in OpenVINO Runtime/CPU: 0.018 seconds per image, FPS: 54.58
PyTorch model on CPU: 0.042 seconds per image, FPS: 24.02
.. parsed-literal::
ONNX model in OpenVINO Runtime/CPU: 0.030 seconds per image, FPS: 33.66
.. parsed-literal::
OpenVINO IR model in OpenVINO Runtime/CPU: 0.029 seconds per image, FPS: 35.01
**Show Device Information**
@ -610,4 +618,4 @@ References
- `Model Conversion API
documentation <https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html>`__
- `Converting Pytorch
model <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_PyTorch.html>`__
model <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch.html>`__

2
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_22_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2c48d5afdee0d49838332deac5fea09e48ea455371a5bc88fd865b070fcffbc3
oid sha256:bdc636d912f3e7146a2aacdaf744a83a90ad66778eacac5924a2499d8acb351f
size 465692

2
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_27_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb50dbc3d8a370e69c72cedea217cb413f04852f64580e0c7b54aa495b6c5ad2
oid sha256:1481c5d43590bbae0644ab25ea265919fb1d8ecd7e4f628e25e2ef69e822c4f0
size 465695

2
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_29_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2c48d5afdee0d49838332deac5fea09e48ea455371a5bc88fd865b070fcffbc3
oid sha256:bdc636d912f3e7146a2aacdaf744a83a90ad66778eacac5924a2499d8acb351f
size 465692

9
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html
View File

@ -1,9 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-onnx-to-openvino-with-output_22_0.png">102-pytorch-onnx-to-openvino-with-output_22_0.png</a> 31-Oct-2023 00:35 465692
<a href="102-pytorch-onnx-to-openvino-with-output_27_0.png">102-pytorch-onnx-to-openvino-with-output_27_0.png</a> 31-Oct-2023 00:35 465695
<a href="102-pytorch-onnx-to-openvino-with-output_29_0.png">102-pytorch-onnx-to-openvino-with-output_29_0.png</a> 31-Oct-2023 00:35 465692
</pre><hr></body>
</html>

9
docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-onnx-to-openvino-with-output_22_0.png">102-pytorch-onnx-to-openvino-with-output_22_0.png</a> 26-Jan-2024 01:05 465692
<a href="102-pytorch-onnx-to-openvino-with-output_27_0.png">102-pytorch-onnx-to-openvino-with-output_27_0.png</a> 26-Jan-2024 01:05 465695
<a href="102-pytorch-onnx-to-openvino-with-output_29_0.png">102-pytorch-onnx-to-openvino-with-output_29_0.png</a> 26-Jan-2024 01:05 465692
</pre><hr></body>
</html>

830
docs/notebooks/102-pytorch-to-openvino-with-output.rst Normal file
View File

@ -0,0 +1,830 @@
Convert a PyTorch Model to OpenVINO™ IR
=======================================
This tutorial demonstrates step-by-step instructions on how to do
inference on a PyTorch classification model using OpenVINO Runtime.
Starting from OpenVINO 2023.0 release, OpenVINO supports direct PyTorch
model conversion without an intermediate step to convert them into ONNX
format. In order, if you try to use the lower OpenVINO version or prefer
to use ONNX, please check this
`tutorial <102-pytorch-to-openvino-with-output.html>`__.
In this tutorial, we will use the
`RegNetY_800MF <https://arxiv.org/abs/2003.13678>`__ model from
`torchvision <https://pytorch.org/vision/stable/index.html>`__ to
demonstrate how to convert PyTorch models to OpenVINO Intermediate
Representation.
The RegNet model was proposed in `Designing Network Design
Spaces <https://arxiv.org/abs/2003.13678>`__ by Ilija Radosavovic, Raj
Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár. The authors
design search spaces to perform Neural Architecture Search (NAS). They
first start from a high dimensional search space and iteratively reduce
the search space by empirically applying constraints based on the
best-performing models sampled by the current search space. Instead of
focusing on designing individual network instances, authors design
network design spaces that parametrize populations of networks. The
overall process is analogous to the classic manual design of networks
but elevated to the design space level. The RegNet design space provides
simple and fast networks that work well across a wide range of flop
regimes.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Load PyTorch Model <#load-pytorch-model>`__
- `Prepare Input Data <#prepare-input-data>`__
- `Run PyTorch Model Inference <#run-pytorch-model-inference>`__
- `Benchmark PyTorch Model
Inference <#benchmark-pytorch-model-inference>`__
- `Convert PyTorch Model to OpenVINO Intermediate
Representation <#convert-pytorch-model-to-openvino-intermediate-representation>`__
- `Select inference device <#select-inference-device>`__
- `Run OpenVINO Model Inference <#run-openvino-model-inference>`__
- `Benchmark OpenVINO Model
Inference <#benchmark-openvino-model-inference>`__
- `Convert PyTorch Model with Static Input
Shape <#convert-pytorch-model-with-static-input-shape>`__
- `Select inference device <#select-inference-device>`__
- `Run OpenVINO Model Inference with Static Input
Shape <#run-openvino-model-inference-with-static-input-shape>`__
- `Benchmark OpenVINO Model Inference with Static Input
Shape <#benchmark-openvino-model-inference-with-static-input-shape>`__
- `Convert TorchScript Model to OpenVINO Intermediate
Representation <#convert-torchscript-model-to-openvino-intermediate-representation>`__
- `Scripted Model <#scripted-model>`__
- `Benchmark Scripted Model
Inference <#benchmark-scripted-model-inference>`__
- `Convert PyTorch Scripted Model to OpenVINO Intermediate
Representation <#convert-pytorch-scripted-model-to-openvino-intermediate-representation>`__
- `Benchmark OpenVINO Model Inference Converted From Scripted
Model <#benchmark-openvino-model-inference-converted-from-scripted-model>`__
- `Traced Model <#traced-model>`__
- `Benchmark Traced Model
Inference <#benchmark-traced-model-inference>`__
- `Convert PyTorch Traced Model to OpenVINO Intermediate
Representation <#convert-pytorch-traced-model-to-openvino-intermediate-representation>`__
- `Benchmark OpenVINO Model Inference Converted From Traced
Model <#benchmark-openvino-model-inference-converted-from-traced-model>`__
Prerequisites
-------------
Install notebook dependencies
.. code:: ipython3
%pip install -q "openvino>=2023.1.0" scipy
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
Download input data and label map
.. code:: ipython3
import requests
from pathlib import Path
from PIL import Image
MODEL_DIR = Path("model")
DATA_DIR = Path("data")
MODEL_DIR.mkdir(exist_ok=True)
DATA_DIR.mkdir(exist_ok=True)
MODEL_NAME = "regnet_y_800mf"
image = Image.open(requests.get("https://farm9.staticflickr.com/8225/8511402100_fea15da1c5_z.jpg", stream=True).raw)
labels_file = DATA_DIR / "imagenet_2012.txt"
if not labels_file.exists():
resp = requests.get("https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/imagenet_2012.txt")
with labels_file.open("wb") as f:
f.write(resp.content)
imagenet_classes = labels_file.open("r").read().splitlines()
Load PyTorch Model
------------------
Generally, PyTorch models represent an instance of the
``torch.nn.Module`` class, initialized by a state dictionary with model
weights. Typical steps for getting a pre-trained model:
1. Create an instance of a model class
2. Load checkpoint state dict, which contains pre-trained model weights
3. Turn the model to evaluation for switching some operations to
inference mode
The ``torchvision`` module provides a ready-to-use set of functions for
model class initialization. We will use
``torchvision.models.regnet_y_800mf``. You can directly pass pre-trained
model weights to the model initialization function using the weights
enum ``RegNet_Y_800MF_Weights.DEFAULT``.
.. code:: ipython3
import torchvision
# get default weights using available weights Enum for model
weights = torchvision.models.RegNet_Y_800MF_Weights.DEFAULT
# create model topology and load weights
model = torchvision.models.regnet_y_800mf(weights=weights)
# switch model to inference mode
model.eval();
Prepare Input Data
~~~~~~~~~~~~~~~~~~
The code below demonstrates how to preprocess input data using a
model-specific transforms module from ``torchvision``. After
transformation, we should concatenate images into batched tensor, in our
case, we will run the model with batch 1, so we just unsqueeze input on
the first dimension.
.. code:: ipython3
import torch
# Initialize the Weight Transforms
preprocess = weights.transforms()
# Apply it to the input image
img_transformed = preprocess(image)
# Add batch dimension to image tensor
input_tensor = img_transformed.unsqueeze(0)
Run PyTorch Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The model returns a vector of probabilities in raw logits format,
softmax can be applied to get normalized values in the [0, 1] range. For
a demonstration that the output of the original model and OpenVINO
converted is the same, we defined a common postprocessing function which
can be reused later.
.. code:: ipython3
import numpy as np
from scipy.special import softmax
# Perform model inference on input tensor
result = model(input_tensor)
# Postprocessing function for getting results in the same way for both PyTorch model inference and OpenVINO
def postprocess_result(output_tensor:np.ndarray, top_k:int = 5):
"""
Posprocess model results. This function applied sofrmax on output tensor and returns specified top_k number of labels with highest probability
Parameters:
output_tensor (np.ndarray): model output tensor with probabilities
top_k (int, *optional*, default 5): number of labels with highest probability for return
Returns:
topk_labels: label ids for selected top_k scores
topk_scores: selected top_k highest scores predicted by model
"""
softmaxed_scores = softmax(output_tensor, -1)[0]
topk_labels = np.argsort(softmaxed_scores)[-top_k:][::-1]
topk_scores = softmaxed_scores[topk_labels]
return topk_labels, topk_scores
# Postprocess results
top_labels, top_scores = postprocess_result(result.detach().numpy())
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_11_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark PyTorch Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
# Run model inference
model(input_tensor)
.. parsed-literal::
16.4 ms ± 673 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model to OpenVINO Intermediate Representation
-------------------------------------------------------------
Starting from the 2023.0 release OpenVINO supports direct PyTorch models
conversion to OpenVINO Intermediate Representation (IR) format. OpenVINO
model conversion API should be used for these purposes. More details
regarding PyTorch model conversion can be found in OpenVINO
`documentation <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch.html>`__
The ``convert_model`` function accepts the PyTorch model object and
returns the ``openvino.Model`` instance ready to load on a device using
``core.compile_model`` or save on disk for next usage using
``ov.save_model``. Optionally, we can provide additional parameters,
such as:
- ``compress_to_fp16`` - flag to perform model weights compression into
FP16 data format. It may reduce the required space for model storage
on disk and give speedup for inference devices, where FP16
calculation is supported.
- ``example_input`` - input data sample which can be used for model
tracing.
- ``input_shape`` - the shape of input tensor for conversion
and any other advanced options supported by model conversion Python API.
More details can be found on this
`page <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__
.. code:: ipython3
import openvino as ov
# Create OpenVINO Core object instance
core = ov.Core()
# Convert model to openvino.runtime.Model object
ov_model = ov.convert_model(model)
# Save openvino.runtime.Model object on disk
ov.save_model(ov_model, MODEL_DIR / f"{MODEL_NAME}_dynamic.xml")
ov_model
.. parsed-literal::
<Model: 'Model30'
inputs[
<ConstOutput: names[x] shape[?,3,?,?] type: f32>
]
outputs[
<ConstOutput: names[x.21] shape[?,1000] type: f32>
]>
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
.. code:: ipython3
# Load OpenVINO model on device
compiled_model = core.compile_model(ov_model, device.value)
compiled_model
.. parsed-literal::
<CompiledModel:
inputs[
<ConstOutput: names[x] shape[?,3,?,?] type: f32>
]
outputs[
<ConstOutput: names[x.21] shape[?,1000] type: f32>
]>
Run OpenVINO Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
# Run model inference
result = compiled_model(input_tensor)[0]
# Posptorcess results
top_labels, top_scores = postprocess_result(result)
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_20_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark OpenVINO Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
compiled_model(input_tensor)
.. parsed-literal::
3.31 ms ± 28.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model with Static Input Shape
---------------------------------------------
The default conversion path preserves dynamic input shapes, in order if
you want to convert the model with static shapes, you can explicitly
specify it during conversion using the ``input_shape`` parameter or
reshape the model into the desired shape after conversion. For the model
reshaping example please check the following
`tutorial <002-openvino-api-with-output.html>`__.
.. code:: ipython3
# Convert model to openvino.runtime.Model object
ov_model = ov.convert_model(model, input=[[1,3,224,224]])
# Save openvino.runtime.Model object on disk
ov.save_model(ov_model, MODEL_DIR / f"{MODEL_NAME}_static.xml")
ov_model
.. parsed-literal::
<Model: 'Model66'
inputs[
<ConstOutput: names[x] shape[1,3,224,224] type: f32>
]
outputs[
<ConstOutput: names[x.21] shape[1,1000] type: f32>
]>
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
.. code:: ipython3
# Load OpenVINO model on device
compiled_model = core.compile_model(ov_model, device.value)
compiled_model
.. parsed-literal::
<CompiledModel:
inputs[
<ConstOutput: names[x] shape[1,3,224,224] type: f32>
]
outputs[
<ConstOutput: names[x.21] shape[1,1000] type: f32>
]>
Now, we can see that input of our converted model is tensor of shape [1,
3, 224, 224] instead of [?, 3, ?, ?] reported by previously converted
model.
Run OpenVINO Model Inference with Static Input Shape
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
# Run model inference
result = compiled_model(input_tensor)[0]
# Posptorcess results
top_labels, top_scores = postprocess_result(result)
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_31_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark OpenVINO Model Inference with Static Input Shape
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
compiled_model(input_tensor)
.. parsed-literal::
2.89 ms ± 38.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert TorchScript Model to OpenVINO Intermediate Representation
-----------------------------------------------------------------
TorchScript is a way to create serializable and optimizable models from
PyTorch code. Any TorchScript program can be saved from a Python process
and loaded in a process where there is no Python dependency. More
details about TorchScript can be found in `PyTorch
documentation <https://pytorch.org/docs/stable/jit.html>`__.
There are 2 possible ways to convert the PyTorch model to TorchScript:
- ``torch.jit.script`` - Scripting a function or ``nn.Module`` will
inspect the source code, compile it as TorchScript code using the
TorchScript compiler, and return a ``ScriptModule`` or
``ScriptFunction``.
- ``torch.jit.trace`` - Trace a function and return an executable or
``ScriptFunction`` that will be optimized using just-in-time
compilation.
Let’s consider both approaches and their conversion into OpenVINO IR.
Scripted Model
~~~~~~~~~~~~~~
``torch.jit.script`` inspects model source code and compiles it to
``ScriptModule``. After compilation model can be used for inference or
saved on disk using the ``torch.jit.save`` function and after that
restored with ``torch.jit.load`` in any other environment without the
original PyTorch model code definitions.
TorchScript itself is a subset of the Python language, so not all
features in Python work, but TorchScript provides enough functionality
to compute on tensors and do control-dependent operations. For a
complete guide, see the `TorchScript Language
Reference <https://pytorch.org/docs/stable/jit_language_reference.html#language-reference>`__.
.. code:: ipython3
# Get model path
scripted_model_path = MODEL_DIR / f"{MODEL_NAME}_scripted.pth"
# Compile and save model if it has not been compiled before or load compiled model
if not scripted_model_path.exists():
scripted_model = torch.jit.script(model)
torch.jit.save(scripted_model, scripted_model_path)
else:
scripted_model = torch.jit.load(scripted_model_path)
# Run scripted model inference
result = scripted_model(input_tensor)
# Postprocess results
top_labels, top_scores = postprocess_result(result.detach().numpy())
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_35_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark Scripted Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
scripted_model(input_tensor)
.. parsed-literal::
12.8 ms ± 6.97 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Scripted Model to OpenVINO Intermediate Representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The conversion step for the scripted model to OpenVINO IR is similar to
the original PyTorch model.
.. code:: ipython3
# Convert model to openvino.runtime.Model object
ov_model = ov.convert_model(scripted_model)
# Load OpenVINO model on device
compiled_model = core.compile_model(ov_model, device.value)
# Run OpenVINO model inference
result = compiled_model(input_tensor, device.value)[0]
# Postprocess results
top_labels, top_scores = postprocess_result(result)
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_39_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark OpenVINO Model Inference Converted From Scripted Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
compiled_model(input_tensor)
.. parsed-literal::
3.41 ms ± 6.84 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Traced Model
~~~~~~~~~~~~
Using ``torch.jit.trace``, you can turn an existing module or Python
function into a TorchScript ``ScriptFunction`` or ``ScriptModule``. You
must provide example inputs, and model will be executed, recording the
operations performed on all the tensors.
- The resulting recording of a standalone function produces
``ScriptFunction``.
- The resulting recording of ``nn.Module.forward`` or ``nn.Module``
produces ``ScriptModule``.
In the same way like scripted model, traced model can be used for
inference or saved on disk using ``torch.jit.save`` function and after
that restored with ``torch.jit.load`` in any other environment without
original PyTorch model code definitions.
.. code:: ipython3
# Get model path
traced_model_path = MODEL_DIR / f"{MODEL_NAME}_traced.pth"
# Trace and save model if it has not been traced before or load traced model
if not traced_model_path.exists():
traced_model = torch.jit.trace(model, example_inputs=input_tensor)
torch.jit.save(traced_model, traced_model_path)
else:
traced_model = torch.jit.load(traced_model_path)
# Run traced model inference
result = traced_model(input_tensor)
# Postprocess results
top_labels, top_scores = postprocess_result(result.detach().numpy())
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_43_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark Traced Model Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
traced_model(input_tensor)
.. parsed-literal::
12.2 ms ± 29 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Traced Model to OpenVINO Intermediate Representation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The conversion step for a traced model to OpenVINO IR is similar to the
original PyTorch model.
.. code:: ipython3
# Convert model to openvino.runtime.Model object
ov_model = ov.convert_model(traced_model)
# Load OpenVINO model on device
compiled_model = core.compile_model(ov_model, device.value)
# Run OpenVINO model inference
result = compiled_model(input_tensor)[0]
# Postprocess results
top_labels, top_scores = postprocess_result(result)
# Show results
display(image)
for idx, (label, score) in enumerate(zip(top_labels, top_scores)):
_, predicted_label = imagenet_classes[label].split(" ", 1)
print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%")
.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_47_0.png
.. parsed-literal::
1: tiger cat - 25.91%
2: Egyptian cat - 10.26%
3: computer keyboard, keypad - 9.22%
4: tabby, tabby cat - 9.09%
5: hamper - 2.35%
Benchmark OpenVINO Model Inference Converted From Traced Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%timeit
compiled_model(input_tensor)[0]
.. parsed-literal::
3.4 ms ± 3.43 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

32
docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html → docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html.tmp
View File

@ -1,20 +1,20 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/102-pytorch-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/102-pytorch-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-to-openvino-with-output_11_0.jpg">102-pytorch-to-openvino-with-output_11_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_11_0.png">102-pytorch-to-openvino-with-output_11_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_20_0.jpg">102-pytorch-to-openvino-with-output_20_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_20_0.png">102-pytorch-to-openvino-with-output_20_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_31_0.jpg">102-pytorch-to-openvino-with-output_31_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_31_0.png">102-pytorch-to-openvino-with-output_31_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_35_0.jpg">102-pytorch-to-openvino-with-output_35_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_35_0.png">102-pytorch-to-openvino-with-output_35_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_39_0.jpg">102-pytorch-to-openvino-with-output_39_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_39_0.png">102-pytorch-to-openvino-with-output_39_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_43_0.jpg">102-pytorch-to-openvino-with-output_43_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_43_0.png">102-pytorch-to-openvino-with-output_43_0.png</a> 07-Dec-2023 00:49 542516
<a href="102-pytorch-to-openvino-with-output_47_0.jpg">102-pytorch-to-openvino-with-output_47_0.jpg</a> 07-Dec-2023 00:49 54874
<a href="102-pytorch-to-openvino-with-output_47_0.png">102-pytorch-to-openvino-with-output_47_0.png</a> 07-Dec-2023 00:49 542516
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="102-pytorch-to-openvino-with-output_11_0.jpg">102-pytorch-to-openvino-with-output_11_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_11_0.png">102-pytorch-to-openvino-with-output_11_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_20_0.jpg">102-pytorch-to-openvino-with-output_20_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_20_0.png">102-pytorch-to-openvino-with-output_20_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_31_0.jpg">102-pytorch-to-openvino-with-output_31_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_31_0.png">102-pytorch-to-openvino-with-output_31_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_35_0.jpg">102-pytorch-to-openvino-with-output_35_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_35_0.png">102-pytorch-to-openvino-with-output_35_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_39_0.jpg">102-pytorch-to-openvino-with-output_39_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_39_0.png">102-pytorch-to-openvino-with-output_39_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_43_0.jpg">102-pytorch-to-openvino-with-output_43_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_43_0.png">102-pytorch-to-openvino-with-output_43_0.png</a> 26-Jan-2024 01:05 542516
<a href="102-pytorch-to-openvino-with-output_47_0.jpg">102-pytorch-to-openvino-with-output_47_0.jpg</a> 26-Jan-2024 01:05 54874
<a href="102-pytorch-to-openvino-with-output_47_0.png">102-pytorch-to-openvino-with-output_47_0.png</a> 26-Jan-2024 01:05 542516
</pre><hr></body>
</html>

72
docs/notebooks/103-paddle-to-openvino-classification-with-output.rst
View File

@ -14,8 +14,8 @@ IR model.
Source of the
`model <https://www.paddlepaddle.org.cn/hubdetail?name=mobilenet_v3_large_imagenet_ssld&en_category=ImageClassification>`__.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Preparation <#preparation>`__
@ -45,28 +45,48 @@ Imports
.. code:: ipython3
%pip install -q "paddlepaddle>=2.5.1"
import platform
if platform.system() == "Windows":
%pip install -q "paddlepaddle>=2.5.1,<2.6.0"
else:
%pip install -q "paddlepaddle>=2.5.1"
%pip install -q paddleclas --no-deps
%pip install -q "prettytable" "ujson" "visualdl>=2.2.0" "faiss-cpu>=1.7.1"
# Install openvino package
!pip install -q "openvino>=2023.1.0"
%pip install -q "openvino>=2023.1.0"
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
paddleclas 2.5.1 requires easydict, which is not installed.
paddleclas 2.5.1 requires faiss-cpu==1.7.1.post2, but you have faiss-cpu 1.7.4 which is incompatible.
paddleclas 2.5.1 requires gast==0.3.3, but you have gast 0.4.0 which is incompatible.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. code:: ipython3
import platform
if (platform.system() == "Linux"):
!wget http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb
!sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb
@ -74,12 +94,20 @@ Imports
.. parsed-literal::
--2023-12-06 22:32:58-- http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb
--2024-01-25 22:35:57-- http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb
Resolving proxy-mu.intel.com (proxy-mu.intel.com)... 10.217.247.236
Connecting to proxy-mu.intel.com (proxy-mu.intel.com)|10.217.247.236|:911... connected.
Proxy request sent, awaiting response... 404 Not Found
2023-12-06 22:32:59 ERROR 404: Not Found.
Proxy request sent, awaiting response...
.. parsed-literal::
404 Not Found
2024-01-25 22:35:57 ERROR 404: Not Found.
.. parsed-literal::
dpkg: error: cannot access archive 'libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb': No such file or directory
@ -107,8 +135,12 @@ Imports
.. parsed-literal::
2023-12-06 22:33:00 INFO: Loading faiss with AVX2 support.
2023-12-06 22:33:00 INFO: Successfully loaded faiss with AVX2 support.
2024-01-25 22:35:59 INFO: Loading faiss with AVX2 support.
.. parsed-literal::
2024-01-25 22:35:59 INFO: Successfully loaded faiss with AVX2 support.
Settings
@ -192,7 +224,11 @@ inference on that image, and then show the top three prediction results.
.. parsed-literal::
[2023/12/06 22:33:21] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead.
[2024/01/25 22:36:20] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead.
.. parsed-literal::
Labrador retriever, 0.75138
German short-haired pointer, 0.02373
Great Dane, 0.01848
@ -201,7 +237,7 @@ inference on that image, and then show the top three prediction results.
.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_1.png
.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_2.png
``classifier.predict()`` takes an image file name, reads the image,
@ -258,7 +294,7 @@ clipping values.
.. parsed-literal::
2023-12-06 22:33:22 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
2024-01-25 22:36:20 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
.. parsed-literal::
@ -270,7 +306,7 @@ clipping values.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7efc106a1910>
<matplotlib.image.AxesImage at 0x7f72946bbac0>
@ -453,7 +489,7 @@ Note that many optimizations are possible to improve the performance.
.. parsed-literal::
PaddlePaddle model on CPU: 0.0070 seconds per image, FPS: 142.41
PaddlePaddle model on CPU: 0.0073 seconds per image, FPS: 137.36
PaddlePaddle result:
Labrador retriever, 0.75138
@ -517,7 +553,7 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0028 seconds per image, FPS: 352.29
OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0031 seconds per image, FPS: 322.19
OpenVINO result:
Labrador retriever, 0.74909
@ -538,4 +574,4 @@ References
- `PaddleClas <https://github.com/PaddlePaddle/PaddleClas>`__
- `OpenVINO PaddlePaddle
support <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_Paddle.html>`__
support <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_Paddle.html>`__

0
docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_1.png → docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_2.png
View File

11
docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html
View File

@ -1,11 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="103-paddle-to-openvino-classification-with-output_15_3.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 07-Dec-2023 00:49 120883
<a href="103-paddle-to-openvino-classification-with-output_23_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 07-Dec-2023 00:49 224886
<a href="103-paddle-to-openvino-classification-with-output_27_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 07-Dec-2023 00:49 224886
<a href="103-paddle-to-openvino-classification-with-output_30_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 07-Dec-2023 00:49 224886
<a href="103-paddle-to-openvino-classification-with-output_8_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 07-Dec-2023 00:49 224886
</pre><hr></body>
</html>

11
docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,11 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="103-paddle-to-openvino-classification-with-output_15_3.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 26-Jan-2024 01:05 120883
<a href="103-paddle-to-openvino-classification-with-output_23_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 26-Jan-2024 01:05 224886
<a href="103-paddle-to-openvino-classification-with-output_27_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 26-Jan-2024 01:05 224886
<a href="103-paddle-to-openvino-classification-with-output_30_1.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 26-Jan-2024 01:05 224886
<a href="103-paddle-to-openvino-classification-with-output_8_2.png">103-paddle-to-openvino-classification-with-outp..&gt;</a> 26-Jan-2024 01:05 224886
</pre><hr></body>
</html>

841
docs/notebooks/104-model-tools-with-output.rst
View File

File diff suppressed because it is too large Load Diff

187
docs/notebooks/105-language-quantize-bert-with-output.rst
View File

@ -22,8 +22,8 @@ and datasets. It consists of the following steps:
- Compare the performance of the original, converted and quantized
models.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Settings <#settings>`__
@ -36,13 +36,13 @@ and datasets. It consists of the following steps:
- `Select inference device <#select-inference-device>`__
- `Compare F1-score of FP32 and INT8
models <#compare-f-score-of-fp-and-int-models>`__
models <#compare-f1-score-of-fp32-and-int8-models>`__
- `Compare Performance of the Original, Converted and Quantized
Models <#compare-performance-of-the-original-converted-and-quantized-models>`__
.. code:: ipython3
%pip install -q "nncf>=2.5.0"
%pip install -q "nncf>=2.5.0"
%pip install -q "transformers" datasets evaluate --extra-index-url https://download.pytorch.org/whl/cpu
%pip install -q "openvino>=2023.1.0"
@ -50,7 +50,15 @@ and datasets. It consists of the following steps:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -67,7 +75,7 @@ Imports
from zipfile import ZipFile
from typing import Iterable
from typing import Any
import datasets
import evaluate
import numpy as np
@ -76,7 +84,7 @@ Imports
import openvino as ov
import torch
from transformers import BertForSequenceClassification, BertTokenizer
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -88,10 +96,14 @@ Imports
.. parsed-literal::
2023-12-06 22:34:55.977192: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-06 22:34:56.010680: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 22:37:51.906403: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 22:37:51.939998: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-06 22:34:56.639162: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 22:37:52.576201: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -112,7 +124,7 @@ Settings
MODEL_LINK = "https://download.pytorch.org/tutorial/MRPC.zip"
FILE_NAME = MODEL_LINK.split("/")[-1]
PRETRAINED_MODEL_DIR = os.path.join(MODEL_DIR, "MRPC")
os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(MODEL_DIR, exist_ok=True)
@ -157,10 +169,10 @@ PyTorch model formats are supported:
input_shape = ov.PartialShape([1, -1])
ir_model_xml = Path(MODEL_DIR) / "bert_mrpc.xml"
core = ov.Core()
torch_model = BertForSequenceClassification.from_pretrained(PRETRAINED_MODEL_DIR)
torch_model.eval
input_info = [("input_ids", input_shape, np.int64),("attention_mask", input_shape, np.int64),("token_type_ids", input_shape, np.int64)]
default_input = torch.ones(1, MAX_SEQ_LENGTH, dtype=torch.int64)
inputs = {
@ -168,7 +180,7 @@ PyTorch model formats are supported:
"attention_mask": default_input,
"token_type_ids": default_input,
}
# Convert the PyTorch model to OpenVINO IR FP32.
if not ir_model_xml.exists():
model = ov.convert_model(torch_model, example_input=inputs, input=input_info)
@ -177,6 +189,12 @@ PyTorch model formats are supported:
model = core.read_model(ir_model_xml)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.__get__(instance, owner)()
.. parsed-literal::
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
@ -185,6 +203,15 @@ PyTorch model formats are supported:
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
.. parsed-literal::
WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.1.0+cpu. If you encounter issues, consider switching to torch==2.1.2
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
@ -203,16 +230,16 @@ tokenizer from HuggingFace.
def create_data_source():
raw_dataset = datasets.load_dataset('glue', 'mrpc', split='validation')
tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_DIR)
def _preprocess_fn(examples):
texts = (examples['sentence1'], examples['sentence2'])
result = tokenizer(*texts, padding='max_length', max_length=MAX_SEQ_LENGTH, truncation=True)
result['labels'] = examples['label']
return result
processed_dataset = raw_dataset.map(_preprocess_fn, batched=True, batch_size=1)
return processed_dataset
data_source = create_data_source()
Optimize model using NNCF Post-training Quantization API
@ -234,7 +261,7 @@ The optimization process contains the following steps:
.. code:: ipython3
INPUT_NAMES = [key for key in inputs.keys()]
def transform_fn(data_item):
"""
Extract the model's input from the data item.
@ -245,7 +272,7 @@ The optimization process contains the following steps:
name: np.asarray([data_item[name]], dtype=np.int64) for name in INPUT_NAMES
}
return inputs
calibration_dataset = nncf.Dataset(data_source, transform_fn)
# Quantize the model. By specifying model_type, we specify additional transformer patterns in the model.
quantized_model = nncf.quantize(model, calibration_dataset,
@ -266,10 +293,6 @@ The optimization process contains the following steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -287,11 +310,6 @@ The optimization process contains the following steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
@ -299,6 +317,11 @@ The optimization process contains the following steps:
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
INFO:nncf:50 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
@ -313,10 +336,6 @@ The optimization process contains the following steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -334,10 +353,6 @@ The optimization process contains the following steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -368,14 +383,14 @@ select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
@ -403,10 +418,10 @@ changing ``sample_idx`` to another value (from 0 to 407).
sample_idx = 5
sample = data_source[sample_idx]
inputs = {k: torch.unsqueeze(torch.tensor(sample[k]), 0) for k in ['input_ids', 'token_type_ids', 'attention_mask']}
result = compiled_quantized_model(inputs)[output_layer]
result = np.argmax(result)
print(f"Text 1: {sample['sentence1']}")
print(f"Text 2: {sample['sentence2']}")
print(f"The same meaning: {'yes' if result == 1 else 'no'}")
@ -428,12 +443,12 @@ Compare F1-score of FP32 and INT8 models
def validate(model: ov.Model, dataset: Iterable[Any]) -> float:
"""
Evaluate the model on GLUE dataset.
Evaluate the model on GLUE dataset.
Returns F1 score metric.
"""
compiled_model = core.compile_model(model, device_name=device.value)
output_layer = compiled_model.output(0)
metric = evaluate.load('glue', 'mrpc')
for batch in dataset:
inputs = [
@ -444,14 +459,14 @@ Compare F1-score of FP32 and INT8 models
metric.add_batch(predictions=[predictions], references=[batch['labels']])
metrics = metric.compute()
f1_score = metrics['f1']
return f1_score
print('Checking the accuracy of the original model:')
metric = validate(model, data_source)
print(f'F1 score: {metric:.4f}')
print('Checking the accuracy of the quantized model:')
metric = validate(quantized_model, data_source)
print(f'F1 score: {metric:.4f}')
@ -460,8 +475,16 @@ Compare F1-score of FP32 and INT8 models
.. parsed-literal::
Checking the accuracy of the original model:
.. parsed-literal::
F1 score: 0.9019
Checking the accuracy of the quantized model:
.. parsed-literal::
F1 score: 0.8969
@ -485,7 +508,7 @@ Frames Per Second (FPS) for images.
num_samples = 50
sample = data_source[0]
inputs = {k: torch.unsqueeze(torch.tensor(sample[k]), 0) for k in ['input_ids', 'token_type_ids', 'attention_mask']}
with torch.no_grad():
start = time.perf_counter()
for _ in range(num_samples):
@ -496,7 +519,7 @@ Frames Per Second (FPS) for images.
f"PyTorch model on CPU: {time_torch / num_samples:.3f} seconds per sentence, "
f"SPS: {num_samples / time_torch:.2f}"
)
start = time.perf_counter()
for _ in range(num_samples):
compiled_model(inputs)
@ -506,7 +529,7 @@ Frames Per Second (FPS) for images.
f"IR FP32 model in OpenVINO Runtime/{device.value}: {time_ir / num_samples:.3f} "
f"seconds per sentence, SPS: {num_samples / time_ir:.2f}"
)
start = time.perf_counter()
for _ in range(num_samples):
compiled_quantized_model(inputs)
@ -525,9 +548,17 @@ Frames Per Second (FPS) for images.
.. parsed-literal::
PyTorch model on CPU: 0.073 seconds per sentence, SPS: 13.77
IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 48.61
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 109.06
PyTorch model on CPU: 0.074 seconds per sentence, SPS: 13.56
.. parsed-literal::
IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 48.16
.. parsed-literal::
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 109.03
Finally, measure the inference performance of OpenVINO ``FP32`` and
@ -557,24 +588,24 @@ in OpenVINO.
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device device.value
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ ERROR ] Exception from src/inference/src/core.cpp:244:
Exception from src/inference/src/dev/core_impl.cpp:559:
[ ERROR ] Exception from src/inference/src/core.cpp:228:
Exception from src/inference/src/dev/core_impl.cpp:560:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 165, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
supported_properties = benchmark.core.get_property(device, properties.supported_properties())
RuntimeError: Exception from src/inference/src/core.cpp:244:
Exception from src/inference/src/dev/core_impl.cpp:559:
RuntimeError: Exception from src/inference/src/core.cpp:228:
Exception from src/inference/src/dev/core_impl.cpp:560:
Device with "device" name is not registered in the OpenVINO Runtime
.. code:: ipython3
@ -590,22 +621,22 @@ in OpenVINO.
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device device.value
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ ERROR ] Exception from src/inference/src/core.cpp:244:
Exception from src/inference/src/dev/core_impl.cpp:559:
[ ERROR ] Exception from src/inference/src/core.cpp:228:
Exception from src/inference/src/dev/core_impl.cpp:560:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 165, in main
supported_properties = benchmark.core.get_property(device, properties.supported_properties())
RuntimeError: Exception from src/inference/src/core.cpp:244:
Exception from src/inference/src/dev/core_impl.cpp:559:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
supported_properties = benchmark.core.get_property(device, properties.supported_properties())
RuntimeError: Exception from src/inference/src/core.cpp:228:
Exception from src/inference/src/dev/core_impl.cpp:560:
Device with "device" name is not registered in the OpenVINO Runtime

219
docs/notebooks/106-auto-device-with-output.rst
View File

@ -13,7 +13,7 @@ network efficiently.
Next, if dedicated accelerators are available, these devices are
preferred (for example, integrated and discrete
`GPU <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-g-p-u>`__).
`GPU <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html>`__).
`CPU <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_CPU.html>`__
is used as the default “fallback device”. Keep in mind that AUTO makes
this selection only once, during the loading of a model.
@ -30,15 +30,13 @@ first inference.
auto
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Import modules and create
Core <#import-modules-and-create-core>`__
- `Import modules and create Core <#import-modules-and-create-core>`__
- `Convert the model to OpenVINO IR
format <#convert-the-model-to-openvino-ir-format>`__
- `(1) Simplify selection
logic <#-simplify-selection-logic>`__
- `(1) Simplify selection logic <#1-simplify-selection-logic>`__
- `Default behavior of Core::compile_model API without
device_name <#default-behavior-of-corecompile_model-api-without-device_name>`__
@ -46,7 +44,7 @@ first inference.
API <#explicitly-pass-auto-as-device_name-to-corecompile_model-api>`__
- `(2) Improve the first inference
latency <#-improve-the-first-inference-latency>`__
latency <#2-improve-the-first-inference-latency>`__
- `Load an Image <#load-an-image>`__
- `Load the model to GPU device and perform
@ -55,19 +53,18 @@ first inference.
inference <#load-the-model-using-auto-device-and-do-inference>`__
- `(3) Achieve different performance for different
targets <#-achieve-different-performance-for-different-targets>`__
targets <#3-achieve-different-performance-for-different-targets>`__
- `Class and callback
definition <#class-and-callback-definition>`__
- `Class and callback definition <#class-and-callback-definition>`__
- `Inference with THROUGHPUT
hint <#inference-with-throughput-hint>`__
- `Inference with LATENCY
hint <#inference-with-latency-hint>`__
- `Difference in FPS and
latency <#difference-in-fps-and-latency>`__
- `Inference with LATENCY hint <#inference-with-latency-hint>`__
- `Difference in FPS and latency <#difference-in-fps-and-latency>`__
Import modules and create Core
------------------------------
Import modules and create Core
------------------------------------------------------------------------
.. code:: ipython3
@ -102,8 +99,10 @@ Import modules and create Core
device to have meaningful results.
Convert the model to OpenVINO IR format
---------------------------------------------------------------------------------
Convert the model to OpenVINO IR format
---------------------------------------
This tutorial uses
`resnet50 <https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html#resnet50>`__
@ -148,11 +147,15 @@ For more information about model conversion API, see this
IR model saved to model/resnet50.xml
(1) Simplify selection logic
----------------------------------------------------------------------
(1) Simplify selection logic
----------------------------
Default behavior of Core::compile_model API without device_name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Default behavior of Core::compile_model API without device_name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By default, ``compile_model`` API will select **AUTO** as
``device_name`` if no device is specified.
@ -171,14 +174,14 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
[22:37:18.2538]I[plugin.cpp:537][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
[22:37:18.2539]I[plugin.cpp:537][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
[22:37:18.2539]I[plugin.cpp:537][AUTO] device:CPU, config:PERF_COUNT=NO
[22:37:18.2539]I[plugin.cpp:542][AUTO] device:CPU, priority:0
[22:37:18.2540]I[schedule.cpp:17][AUTO] scheduler starting
[22:37:18.2540]I[auto_schedule.cpp:131][AUTO] select device:CPU
[22:37:18.3716]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
[22:37:18.3717]I[plugin.cpp:572][AUTO] underlying hardware does not support hardware context
[22:41:12.8903]I[plugin.cpp:536][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
[22:41:12.8903]I[plugin.cpp:536][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
[22:41:12.8904]I[plugin.cpp:536][AUTO] device:CPU, config:PERF_COUNT=NO
[22:41:12.8904]I[plugin.cpp:541][AUTO] device:CPU, priority:0
[22:41:12.8904]I[schedule.cpp:17][AUTO] scheduler starting
[22:41:12.8904]I[auto_schedule.cpp:131][AUTO] select device:CPU
[22:41:13.0500]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
[22:41:13.0502]I[plugin.cpp:569][AUTO] underlying hardware does not support hardware context
Successfully compiled model without a device_name.
@ -191,12 +194,14 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
[22:37:18.3836]I[schedule.cpp:303][AUTO] scheduler ending
Deleted compiled_model
[22:41:13.0623]I[schedule.cpp:303][AUTO] scheduler ending
Explicitly pass AUTO as device_name to Core::compile_model API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Explicitly pass AUTO as device_name to Core::compile_model API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is optional, but passing AUTO explicitly as ``device_name`` may
improve readability of your code.
@ -229,8 +234,10 @@ improve readability of your code.
Deleted compiled_model
(2) Improve the first inference latency
---------------------------------------------------------------------------------
(2) Improve the first inference latency
---------------------------------------
One of the benefits of using AUTO device selection is reducing FIL
(first inference latency). FIL is the model compilation time combined
@ -243,8 +250,10 @@ This initialization time may be intolerable for some applications. To
avoid this delay, the AUTO uses CPU transparently as the first inference
device until GPU is ready.
Load an Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load an Image
~~~~~~~~~~~~~
torchvision library provides model specific input transformation
function, we will reuse it for preparing input data.
@ -289,8 +298,10 @@ function, we will reuse it for preparing input data.
Load the model to GPU device and perform inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the model to GPU device and perform inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -316,8 +327,10 @@ Load the model to GPU device and perform inference
A GPU device is not available. Available devices are: ['CPU']
Load the model using AUTO device and do inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the model using AUTO device and do inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When GPU is the best available device, the first few inferences will be
executed on CPU until GPU is ready.
@ -340,7 +353,7 @@ executed on CPU until GPU is ready.
.. parsed-literal::
Time to load model using AUTO device and get first inference: 0.14 seconds.
Time to load model using AUTO device and get first inference: 0.15 seconds.
.. code:: ipython3
@ -348,8 +361,10 @@ executed on CPU until GPU is ready.
# Deleted model will wait for compiling on the selected device to complete.
del compiled_model
(3) Achieve different performance for different targets
-------------------------------------------------------------------------------------------------
(3) Achieve different performance for different targets
-------------------------------------------------------
It is an advantage to define **performance hints** when using Automatic
Device Selection. By specifying a **THROUGHPUT** or **LATENCY** hint,
@ -361,13 +376,15 @@ completely portable between devices – meaning AUTO can configure the
performance hint on whichever device is being used.
For more information, refer to the `Performance
Hints <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html#performance-hints>`__
Hints <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html#performance-hints-for-auto>`__
section of `Automatic Device
Selection <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
article.
Class and callback definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Class and callback definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -465,8 +482,10 @@ Class and callback definition
metrics_update_interval = 10
metrics_update_num = 6
Inference with THROUGHPUT hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inference with THROUGHPUT hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Loop for inference and update the FPS/Latency every
@metrics_update_interval seconds.
@ -503,18 +522,52 @@ Loop for inference and update the FPS/Latency every
.. parsed-literal::
Compiling Model for AUTO device with THROUGHPUT hint
.. parsed-literal::
Start inference, 6 groups of FPS/latency will be measured over 10s intervals
throughput: 182.05fps, latency: 31.31ms, time interval: 10.01s
throughput: 182.68fps, latency: 32.06ms, time interval: 10.01s
throughput: 183.56fps, latency: 31.93ms, time interval: 10.02s
throughput: 182.77fps, latency: 32.06ms, time interval: 10.01s
throughput: 182.17fps, latency: 32.13ms, time interval: 10.00s
throughput: 182.39fps, latency: 32.15ms, time interval: 10.00s
.. parsed-literal::
throughput: 184.87fps, latency: 30.81ms, time interval: 10.01s
.. parsed-literal::
throughput: 185.21fps, latency: 31.62ms, time interval: 10.02s
.. parsed-literal::
throughput: 185.47fps, latency: 31.54ms, time interval: 10.00s
.. parsed-literal::
throughput: 185.75fps, latency: 31.53ms, time interval: 10.01s
.. parsed-literal::
throughput: 185.09fps, latency: 31.64ms, time interval: 10.00s
.. parsed-literal::
throughput: 184.42fps, latency: 31.76ms, time interval: 10.00s
.. parsed-literal::
Done
Inference with LATENCY hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inference with LATENCY hint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Loop for inference and update the FPS/Latency for each
@metrics_update_interval seconds
@ -552,18 +605,52 @@ Loop for inference and update the FPS/Latency for each
.. parsed-literal::
Compiling Model for AUTO Device with LATENCY hint
.. parsed-literal::
Start inference, 6 groups fps/latency will be out with 10s interval
throughput: 140.29fps, latency: 6.65ms, time interval: 10.00s
throughput: 142.87fps, latency: 6.63ms, time interval: 10.00s
throughput: 142.41fps, latency: 6.64ms, time interval: 10.01s
throughput: 142.91fps, latency: 6.63ms, time interval: 10.01s
throughput: 142.78fps, latency: 6.64ms, time interval: 10.00s
throughput: 142.80fps, latency: 6.64ms, time interval: 10.01s
.. parsed-literal::
throughput: 138.57fps, latency: 6.67ms, time interval: 10.00s
.. parsed-literal::
throughput: 141.04fps, latency: 6.66ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.74fps, latency: 6.66ms, time interval: 10.00s
.. parsed-literal::
throughput: 141.60fps, latency: 6.68ms, time interval: 10.01s
.. parsed-literal::
throughput: 141.73fps, latency: 6.68ms, time interval: 10.00s
.. parsed-literal::
throughput: 141.45fps, latency: 6.67ms, time interval: 10.00s
.. parsed-literal::
Done
Difference in FPS and latency
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Difference in FPS and latency
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3

4
docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_27_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1f838f2aa8318b73e4dadf5d48bbfb1332a4bebc2ca4d854dd81e77c93b5b244
size 26784
oid sha256:1b8fb134deae365a3c5b358b032e2ad1e5ccaccae51cb176f250ba946461d8d6
size 25724

4
docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_28_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8187fa0479ea8f4d92a4b1cb42aa15f5377ce2630fa924bee73168ba2c5d10b4
size 39984
oid sha256:d311a41fc8902df2f1ca223ffcdcf1ff634d160376b890b3a571326912b1d88a
size 40017

10
docs/notebooks/106-auto-device-with-output_files/index.html
View File

@ -1,10 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/106-auto-device-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/106-auto-device-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="106-auto-device-with-output_14_1.jpg">106-auto-device-with-output_14_1.jpg</a> 31-Oct-2023 00:35 121563
<a href="106-auto-device-with-output_14_1.png">106-auto-device-with-output_14_1.png</a> 31-Oct-2023 00:35 869661
<a href="106-auto-device-with-output_27_0.png">106-auto-device-with-output_27_0.png</a> 31-Oct-2023 00:35 26784
<a href="106-auto-device-with-output_28_0.png">106-auto-device-with-output_28_0.png</a> 31-Oct-2023 00:35 39984
</pre><hr></body>
</html>

10
docs/notebooks/106-auto-device-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/106-auto-device-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/106-auto-device-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="106-auto-device-with-output_14_1.jpg">106-auto-device-with-output_14_1.jpg</a> 26-Jan-2024 01:05 121563
<a href="106-auto-device-with-output_14_1.png">106-auto-device-with-output_14_1.png</a> 26-Jan-2024 01:05 869661
<a href="106-auto-device-with-output_27_0.png">106-auto-device-with-output_27_0.png</a> 26-Jan-2024 01:05 25724
<a href="106-auto-device-with-output_28_0.png">106-auto-device-with-output_28_0.png</a> 26-Jan-2024 01:05 40017
</pre><hr></body>
</html>

438
docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst
View File

File diff suppressed because one or more lines are too long

833
docs/notebooks/107-speech-recognition-quantization-wav2vec2-with-output.rst Normal file
View File

File diff suppressed because one or more lines are too long

347
docs/notebooks/108-gpu-device-with-output.rst
View File

@ -1,8 +1,8 @@
Working with GPUs in OpenVINO™
==============================
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Introduction <#introduction>`__
@ -12,16 +12,15 @@ Working with GPUs in OpenVINO™
Device <#checking-gpus-with-query-device>`__
- `List GPUs with
core.available_devices <#list-gpus-with-coreavailable_devices>`__
core.available_devices <#list-gpus-with-core-available_devices>`__
- `Check Properties with
core.get_property <#check-properties-with-coreget_property>`__
core.get_property <#check-properties-with-core-get_property>`__
- `Brief Descriptions of Key
Properties <#brief-descriptions-of-key-properties>`__
- `Compiling a Model on GPU <#compiling-a-model-on-gpu>`__
- `Download and Convert a
Model <#download-and-convert-a-model>`__
- `Download and Convert a Model <#download-and-convert-a-model>`__
- `Download and unpack the
Model <#download-and-unpack-the-model>`__
@ -39,22 +38,20 @@ Working with GPUs in OpenVINO™
- `Performance Comparison with
benchmark_app <#performance-comparison-with-benchmark_app>`__
- `CPU vs GPU with Latency
Hint <#cpu-vs-gpu-with-latency-hint>`__ - `CPU vs GPU with
Throughput Hint <#cpu-vs-gpu-with-throughput-hint>`__ -
`Single GPU vs Multiple
GPUs <#single-gpu-vs-multiple-gpus>`__
- `Basic Application Using
GPUs <#basic-application-using-gpus>`__
- `CPU vs GPU with Latency Hint <#cpu-vs-gpu-with-latency-hint>`__
- `CPU vs GPU with Throughput
Hint <#cpu-vs-gpu-with-throughput-hint>`__
- `Single GPU vs Multiple GPUs <#single-gpu-vs-multiple-gpus>`__
- `Basic Application Using GPUs <#basic-application-using-gpus>`__
- `Import Necessary Packages <#import-necessary-packages>`__
- `Compile the Model <#compile-the-model>`__
- `Load and Preprocess Video
Frames <#load-and-preprocess-video-frames>`__
- `Define Model Output
Classes <#define-model-output-classes>`__
- `Set up Asynchronous
Pipeline <#set-up-asynchronous-pipeline>`__
- `Define Model Output Classes <#define-model-output-classes>`__
- `Set up Asynchronous Pipeline <#set-up-asynchronous-pipeline>`__
- `Callback Definition <#callback-definition>`__
- `Create Async Pipeline <#create-async-pipeline>`__
@ -76,7 +73,9 @@ provides the code for a basic end-to-end application that compiles a
model on GPU and uses it to run inference.
Introduction
------------------------------------------------------
------------
Originally, graphic processing units (GPUs) began as specialized chips,
developed to accelerate the rendering of computer graphics. In contrast
@ -103,13 +102,15 @@ to configure OpenVINO to work with your GPU. Then, read on to learn how
to accelerate inference with GPUs in OpenVINO!
Install required packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%pip install -q "openvino-dev>=2023.1.0"
%pip install -q tensorflow
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -127,13 +128,17 @@ Install required packages
Checking GPUs with Query Device
-------------------------------------------------------------------------
-------------------------------
In this section, we will see how to list the available GPUs and check
their properties. Some of the key properties will also be defined.
List GPUs with core.available_devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO Runtime provides the ``available_devices`` method for checking
which devices are available for inference. The following code will
@ -143,7 +148,7 @@ appear.
.. code:: ipython3
import openvino as ov
core = ov.Core()
core.available_devices
@ -172,7 +177,9 @@ to configure your GPU drivers to work with OpenVINO. Once we have the
GPUs working with OpenVINO, we can proceed with the next sections.
Check Properties with core.get_property
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get information about the GPUs, we can use device properties. In
OpenVINO, devices have properties that describe their characteristics
@ -185,7 +192,7 @@ To get the value of a property, such as the device name, we can use the
.. code:: ipython3
device = "GPU"
core.get_property(device, "FULL_DEVICE_NAME")
@ -209,7 +216,7 @@ for that property.
print(f"{device} SUPPORTED_PROPERTIES:\n")
supported_properties = core.get_property(device, "SUPPORTED_PROPERTIES")
indent = len(max(supported_properties, key=len))
for property_key in supported_properties:
if property_key not in ('SUPPORTED_METRICS', 'SUPPORTED_CONFIG_KEYS', 'SUPPORTED_PROPERTIES'):
try:
@ -222,7 +229,7 @@ for that property.
.. parsed-literal::
GPU SUPPORTED_PROPERTIES:
AVAILABLE_DEVICES : ['0']
RANGE_FOR_ASYNC_INFER_REQUESTS: (1, 2, 1)
RANGE_FOR_STREAMS : (1, 2)
@ -245,8 +252,8 @@ for that property.
GPU_QUEUE_PRIORITY : Priority.MEDIUM
GPU_QUEUE_THROTTLE : Priority.MEDIUM
GPU_ENABLE_LOOP_UNROLLING : True
CACHE_DIR :
PERFORMANCE_HINT : PerformanceMode.LATENCY
CACHE_DIR :
PERFORMANCE_HINT : PerformanceMode.UNDEFINED
COMPILATION_NUM_THREADS : 20
NUM_STREAMS : 1
PERFORMANCE_HINT_NUM_REQUESTS : 0
@ -255,7 +262,9 @@ for that property.
Brief Descriptions of Key Properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each device has several properties as seen in the last command. Some of
the key properties are:
@ -283,7 +292,9 @@ Properties <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_query_api.html>`
page.
Compiling a Model on GPU
------------------------------------------------------------------
------------------------
Now, we know how to list the GPUs in the system and check their
properties. We can easily use one for compiling and running models with
@ -291,7 +302,9 @@ OpenVINO `GPU
plugin <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html>`__.
Download and Convert a Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This tutorial uses the ``ssdlite_mobilenet_v2`` model. The
``ssdlite_mobilenet_v2`` model is used for object detection. The model
@ -301,7 +314,9 @@ categories of object. For details, see the
`paper <https://arxiv.org/abs/1801.04381>`__.
Download and unpack the Model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use the ``download_file`` function from the ``notebook_utils`` to
download an archive with the model. It automatically creates a directory
@ -313,23 +328,23 @@ package is already downloaded.
import sys
import tarfile
from pathlib import Path
sys.path.append("../utils")
import notebook_utils as utils
# A directory where the model will be downloaded.
base_model_dir = Path("./model").expanduser()
model_name = "ssdlite_mobilenet_v2"
archive_name = Path(f"{model_name}_coco_2018_05_09.tar.gz")
# Download the archive
downloaded_model_path = base_model_dir / archive_name
if not downloaded_model_path.exists():
model_url = f"http://download.tensorflow.org/models/object_detection/{archive_name}"
utils.download_file(model_url, downloaded_model_path.name, downloaded_model_path.parent)
# Unpack the model
tf_model_path = base_model_dir / archive_name.with_suffix("").stem / "frozen_inference_graph.pb"
if not tf_model_path.exists():
@ -350,15 +365,17 @@ package is already downloaded.
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.
Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)
Convert the Model to OpenVINO IR format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To convert the model to OpenVINO IR with ``FP16`` precision, use model
conversion API. The models are saved to the ``model/ir_model/``
@ -368,15 +385,15 @@ directory. For more details about model conversion, see this
.. code:: ipython3
from openvino.tools.mo.front import tf as ov_tf_front
precision = 'FP16'
# The output path for the conversion.
model_path = base_model_dir / 'ir_model' / f'{model_name}_{precision.lower()}.xml'
trans_config_path = Path(ov_tf_front.__file__).parent / "ssd_v2_support.json"
pipeline_config = base_model_dir / archive_name.with_suffix("").stem / "pipeline.config"
model = None
if not model_path.exists():
model = ov.tools.mo.convert_model(input_model=tf_model_path,
@ -403,7 +420,9 @@ directory. For more details about model conversion, see this
Compile with Default Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When the model is ready, first we need to read it, using the
``read_model`` method. Then, we can use the ``compile_model`` method and
@ -425,7 +444,9 @@ page as well as the `AUTO device
tutorial <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/106-auto-device>`__.
Reduce Compile Time through Model Caching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Depending on the model used, device-specific optimizations and network
compilations can cause the compile step to be time-consuming, especially
@ -440,17 +461,17 @@ following:
import time
from pathlib import Path
# Create cache folder
cache_folder = Path("cache")
cache_folder.mkdir(exist_ok=True)
start = time.time()
core = Core()
core = ov.Core()
# Set cache folder
core.set_property({'CACHE_DIR': cache_folder})
# Compile the model as before
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
@ -468,14 +489,14 @@ compile times with caching enabled and disabled as follows:
.. code:: ipython3
start = time.time()
core = Core()
core = ov.Core()
core.set_property({'CACHE_DIR': 'cache'})
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"Cache enabled - compile time: {time.time() - start}s")
start = time.time()
core = Core()
core = ov.Core()
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"Cache disabled - compile time: {time.time() - start}s")
@ -494,7 +515,9 @@ Caching <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Model_caching_overv
docs.
Throughput and Latency Performance Hints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To simplify device and pipeline configuration, OpenVINO provides
high-level performance hints that automatically set the batch size and
@ -524,7 +547,9 @@ available memory.
compiled_model = core.compile_model(model, device, {"PERFORMANCE_HINT": "THROUGHPUT"})
Using Multiple GPUs with Multi-Device and Cumulative Throughput
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The latency and throughput hints mentioned above are great and can make
a difference when used adequately but they usually use just one device,
@ -566,7 +591,9 @@ manually specify devices to use. Below is an example showing how to use
notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/115-async-api>`__.
Performance Comparison with benchmark_app
-----------------------------------------------------------------------------------
-----------------------------------------
Given all the different options available when compiling a model, it may
be difficult to know which settings work best for a certain application.
@ -601,12 +628,12 @@ with a latency focus:
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
@ -635,7 +662,7 @@ with a latency focus:
[ INFO ] GPU_QUEUE_PRIORITY: Priority.MEDIUM
[ INFO ] GPU_QUEUE_THROTTLE: Priority.MEDIUM
[ INFO ] GPU_ENABLE_LOOP_UNROLLING: True
[ INFO ] CACHE_DIR:
[ INFO ] CACHE_DIR:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] COMPILATION_NUM_THREADS: 20
[ INFO ] NUM_STREAMS: 1
@ -644,7 +671,7 @@ with a latency focus:
[ INFO ] DEVICE_ID: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'image_tensor'!. This input will be filled with random values!
[ INFO ] Fill input 'image_tensor' with random values
[ INFO ] Fill input 'image_tensor' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 6.17 ms
@ -666,7 +693,9 @@ GPU to be better than CPU, whereas multiple GPUs should be better than a
single GPU as long as there is enough work for each of them.
CPU vs GPU with Latency Hint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
@ -680,12 +709,12 @@ CPU vs GPU with Latency Hint
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
@ -717,7 +746,7 @@ CPU vs GPU with Latency Hint
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'image_tensor'!. This input will be filled with random values!
[ INFO ] Fill input 'image_tensor' with random values
[ INFO ] Fill input 'image_tensor' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 4.42 ms
@ -744,12 +773,12 @@ CPU vs GPU with Latency Hint
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
@ -778,7 +807,7 @@ CPU vs GPU with Latency Hint
[ INFO ] GPU_QUEUE_PRIORITY: Priority.MEDIUM
[ INFO ] GPU_QUEUE_THROTTLE: Priority.MEDIUM
[ INFO ] GPU_ENABLE_LOOP_UNROLLING: True
[ INFO ] CACHE_DIR:
[ INFO ] CACHE_DIR:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] COMPILATION_NUM_THREADS: 20
[ INFO ] NUM_STREAMS: 1
@ -787,7 +816,7 @@ CPU vs GPU with Latency Hint
[ INFO ] DEVICE_ID: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'image_tensor'!. This input will be filled with random values!
[ INFO ] Fill input 'image_tensor' with random values
[ INFO ] Fill input 'image_tensor' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 8.79 ms
@ -803,7 +832,9 @@ CPU vs GPU with Latency Hint
CPU vs GPU with Throughput Hint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
@ -817,12 +848,12 @@ CPU vs GPU with Throughput Hint
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
@ -854,7 +885,7 @@ CPU vs GPU with Throughput Hint
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'image_tensor'!. This input will be filled with random values!
[ INFO ] Fill input 'image_tensor' with random values
[ INFO ] Fill input 'image_tensor' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 5 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 8.15 ms
@ -881,12 +912,12 @@ CPU vs GPU with Throughput Hint
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
@ -915,7 +946,7 @@ CPU vs GPU with Throughput Hint
[ INFO ] GPU_QUEUE_PRIORITY: Priority.MEDIUM
[ INFO ] GPU_QUEUE_THROTTLE: Priority.MEDIUM
[ INFO ] GPU_ENABLE_LOOP_UNROLLING: True
[ INFO ] CACHE_DIR:
[ INFO ] CACHE_DIR:
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] COMPILATION_NUM_THREADS: 20
[ INFO ] NUM_STREAMS: 2
@ -924,7 +955,7 @@ CPU vs GPU with Throughput Hint
[ INFO ] DEVICE_ID: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'image_tensor'!. This input will be filled with random values!
[ INFO ] Fill input 'image_tensor' with random values
[ INFO ] Fill input 'image_tensor' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 9.17 ms
@ -940,7 +971,9 @@ CPU vs GPU with Throughput Hint
Single GPU vs Multiple GPUs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
@ -954,12 +987,12 @@ Single GPU vs Multiple GPUs
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Device GPU.1 does not support performance hint property(-hint).
[ ERROR ] Config for device with 1 ID is not registered in GPU plugin
@ -983,14 +1016,14 @@ Single GPU vs Multiple GPUs
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Device GPU.1 does not support performance hint property(-hint).
[Step 4/11] Reading model files
@ -1030,14 +1063,14 @@ Single GPU vs Multiple GPUs
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ] MULTI
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Device GPU.1 does not support performance hint property(-hint).
[Step 4/11] Reading model files
@ -1066,7 +1099,9 @@ Single GPU vs Multiple GPUs
Basic Application Using GPUs
----------------------------------------------------------------------
----------------------------
We will now show an end-to-end object detection example using GPUs in
OpenVINO. The application compiles a model on GPU with the “THROUGHPUT”
@ -1078,20 +1113,22 @@ corresponding frame and saved as a video, which is displayed at the end
of the application.
Import Necessary Packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
import time
from pathlib import Path
import cv2
import numpy as np
from IPython.display import Video
from openvino.runtime import AsyncInferQueue, Core, InferRequest
import openvino as ov
# Instantiate OpenVINO Runtime
core = Core()
core = ov.Core()
core.available_devices
@ -1104,7 +1141,9 @@ Import Necessary Packages
Compile the Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -1112,11 +1151,11 @@ Compile the Model
model = core.read_model(model=model_path)
device_name = "GPU"
compiled_model = core.compile_model(model=model, device_name=device_name, config={"PERFORMANCE_HINT": "THROUGHPUT"})
# Get the input and output nodes
input_layer = compiled_model.input(0)
output_layer = compiled_model.output(0)
# Get the input size
num, height, width, channels = input_layer.shape
print('Model input shape:', num, height, width, channels)
@ -1128,7 +1167,9 @@ Compile the Model
Load and Preprocess Video Frames
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -1136,7 +1177,7 @@ Load and Preprocess Video Frames
video_file = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/Coco%20Walking%20in%20Berkeley.mp4"
video = cv2.VideoCapture(video_file)
framebuf = []
# Go through every frame of video and resize it
print('Loading video...')
while video.isOpened():
@ -1145,18 +1186,18 @@ Load and Preprocess Video Frames
print('Video loaded!')
video.release()
break
# Preprocess frames - convert them to shape expected by model
input_frame = cv2.resize(src=frame, dsize=(width, height), interpolation=cv2.INTER_AREA)
input_frame = np.expand_dims(input_frame, axis=0)
# Append frame to framebuffer
framebuf.append(input_frame)
print('Frame shape: ', framebuf[0].shape)
print('Number of frames: ', len(framebuf))
# Show original video file
# If the video does not display correctly inside the notebook, please open it with your favorite media player
Video(video_file)
@ -1171,7 +1212,9 @@ Load and Preprocess Video Frames
Define Model Output Classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -1192,36 +1235,44 @@ Define Model Output Classes
]
Set up Asynchronous Pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Callback Definition
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
# Define a callback function that runs every time the asynchronous pipeline completes inference on a frame
def completion_callback(infer_request: InferRequest, frame_id: int) -> None:
def completion_callback(infer_request: ov.InferRequest, frame_id: int) -> None:
global frame_number
stop_time = time.time()
frame_number += 1
predictions = next(iter(infer_request.results.values()))
results[frame_id] = predictions[:10] # Grab first 10 predictions for this frame
total_time = stop_time - start_time
frame_fps[frame_id] = frame_number / total_time
Create Async Pipeline
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
# Create asynchronous inference queue with optimal number of infer requests
infer_queue = AsyncInferQueue(compiled_model)
infer_queue = ov.AsyncInferQueue(compiled_model)
infer_queue.set_callback(completion_callback)
Perform Inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -1232,14 +1283,14 @@ Perform Inference
start_time = time.time()
for i, input_frame in enumerate(framebuf):
infer_queue.start_async({0: input_frame}, i)
infer_queue.wait_all() # Wait until all inference requests in the AsyncInferQueue are completed
stop_time = time.time()
# Calculate total inference time and FPS
total_time = stop_time - start_time
fps = len(framebuf) / total_time
time_per_frame = 1 / fps
time_per_frame = 1 / fps
print(f'Total time to infer all frames: {total_time:.3f}s')
print(f'Time per frame: {time_per_frame:.6f}s ({fps:.3f} FPS)')
@ -1251,26 +1302,28 @@ Perform Inference
Process Results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~
.. code:: ipython3
# Set minimum detection threshold
min_thresh = .6
# Load video
video = cv2.VideoCapture(video_file)
# Get video parameters
frame_width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = int(video.get(cv2.CAP_PROP_FPS))
fourcc = int(video.get(cv2.CAP_PROP_FOURCC))
# Create folder and VideoWriter to save output video
Path('./output').mkdir(exist_ok=True)
output = cv2.VideoWriter('output/output.mp4', fourcc, fps, (frame_width, frame_height))
# Draw detection results on every frame of video and save as a new video file
while video.isOpened():
current_frame = int(video.get(cv2.CAP_PROP_POS_FRAMES))
@ -1280,12 +1333,12 @@ Process Results
output.release()
video.release()
break
# Draw info at the top left such as current fps, the devices and the performance hint being used
cv2.putText(frame, f"fps {str(round(frame_fps[current_frame], 2))}", (5, 20), cv2.FONT_ITALIC, 0.6, (0, 0, 0), 1, cv2.LINE_AA)
cv2.putText(frame, f"device {device_name}", (5, 40), cv2.FONT_ITALIC, 0.6, (0, 0, 0), 1, cv2.LINE_AA)
cv2.putText(frame, f"device {device_name}", (5, 40), cv2.FONT_ITALIC, 0.6, (0, 0, 0), 1, cv2.LINE_AA)
cv2.putText(frame, f"hint {compiled_model.get_property('PERFORMANCE_HINT').name}", (5, 60), cv2.FONT_ITALIC, 0.6, (0, 0, 0), 1, cv2.LINE_AA)
# prediction contains [image_id, label, conf, x_min, y_min, x_max, y_max] according to model
for prediction in np.squeeze(results[current_frame]):
if prediction[2] > min_thresh:
@ -1294,13 +1347,13 @@ Process Results
x_max = int(prediction[5] * frame_width)
y_max = int(prediction[6] * frame_height)
label = classes[int(prediction[1])]
# Draw a bounding box with its label above it
cv2.rectangle(frame, (x_min, y_min), (x_max, y_max), (0, 255, 0), 1, cv2.LINE_AA)
cv2.putText(frame, label, (x_min, y_min - 10), cv2.FONT_ITALIC, 1, (255, 0, 0), 1, cv2.LINE_AA)
output.write(frame)
# Show output video file
# If the video does not display correctly inside the notebook, please open it with your favorite media player
Video("output/output.mp4", width=800, embed=True)
@ -1323,7 +1376,9 @@ Process Results
Conclusion
----------------------------------------------------
----------
This tutorial demonstrates how easy it is to use one or more GPUs in
OpenVINO, check their properties, and even tailor the model performance
@ -1334,11 +1389,19 @@ detected bounding boxes.
To read more about any of these topics, feel free to visit their
corresponding documentation:
- `GPU Plugin <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html>`__
- `AUTO Plugin <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
- `Model Caching <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Model_caching_overview.html>`__
- `MULTI Device Mode <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_Running_on_multiple_devices.html>`__
- `Query Device Properties <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_query_api.html>`__
- `Configurations for GPUs with OpenVINO <https://docs.openvino.ai/2023.3/openvino_docs_install_guides_configurations_for_intel_gpu.html>`__
- `Benchmark Python Tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html>`__
- `Asynchronous Inferencing <https://docs.openvino.ai/2023.3/openvino_docs_ov_plugin_dg_async_infer_request.html>`__
- `GPU
Plugin <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html>`__
- `AUTO
Plugin <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html>`__
- `Model
Caching <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Model_caching_overview.html>`__
- `MULTI Device
Mode <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_Running_on_multiple_devices.html>`__
- `Query Device
Properties <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_query_api.html>`__
- `Configurations for GPUs with
OpenVINO <https://docs.openvino.ai/2023.3/openvino_docs_install_guides_configurations_for_intel_gpu.html>`__
- `Benchmark Python
Tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html>`__
- `Asynchronous
Inferencing <https://docs.openvino.ai/2023.3/openvino_docs_ov_plugin_dg_async_infer_request.html>`__

255
docs/notebooks/109-latency-tricks-with-output.rst
View File

@ -27,8 +27,8 @@ The quantization and pre-post-processing API are not included here as
they change the precision (quantization) or processing graph
(prepostprocessor). You can find examples of how to apply them to
optimize performance on OpenVINO IR files in
`111-detection-quantization <../111-detection-quantization>`__ and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
`111-detection-quantization <111-yolov5-quantization-migration-with-output.html>`__ and
`118-optimize-preprocessing <118-optimize-preprocessing-with-output.html>`__.
|image0|
@ -44,10 +44,10 @@ optimize performance on OpenVINO IR files in
result in different performance.
A similar notebook focused on the throughput mode is available
`here <109-throughput-tricks.ipynb>`__.
**Table of contents:**
`here <109-throughput-tricks-with-output.html>`__.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Data <#data>`__
@ -94,7 +94,7 @@ Prerequisites
import time
from pathlib import Path
from typing import Any, List, Tuple
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -116,19 +116,19 @@ requirements of this particular object detection model.
import numpy as np
import cv2
IMAGE_WIDTH = 640
IMAGE_HEIGHT = 480
# load image
image = utils.load_image("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg")
image = cv2.resize(image, dsize=(IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_AREA)
# preprocess it for YOLOv5
input_image = image / 255.0
input_image = np.transpose(input_image, axes=(2, 0, 1))
input_image = np.expand_dims(input_image, axis=0)
# show the image
utils.show_array(image)
@ -141,7 +141,7 @@ requirements of this particular object detection model.
.. parsed-literal::
<DisplayHandle display_id=e5b567bdbd103853038ec2c801bd914a>
<DisplayHandle display_id=b3ed024cde96c857177f3da66878c56a>
@ -159,13 +159,13 @@ PyTorch Hub and small enough to see the difference in performance.
import torch
from IPython.utils import io
# directory for all models
base_model_dir = Path("model")
model_name = "yolov5n"
model_path = base_model_dir / model_name
# load YOLOv5n from PyTorch Hub
pytorch_model = torch.hub.load("ultralytics/yolov5", "custom", path=model_path, device="cpu", skip_validation=True)
# don't print full model architecture
@ -176,8 +176,12 @@ PyTorch Hub and small enough to see the difference in performance.
.. parsed-literal::
Using cache found in /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master
.. parsed-literal::
YOLOv5 🚀 2023-4-21 Python-3.8.10 torch-2.1.0+cpu CPU
.. parsed-literal::
@ -188,11 +192,86 @@ PyTorch Hub and small enough to see the difference in performance.
.. parsed-literal::
Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5n.pt to model/yolov5n.pt...
100%|██████████| 3.87M/3.87M [00:02<00:00, 1.50MB/s]
Fusing layers...
.. parsed-literal::
0%| | 0.00/3.87M [00:00<?, ?B/s]
.. parsed-literal::
5%|▌ | 208k/3.87M [00:00<00:01, 2.11MB/s]
.. parsed-literal::
15%|█▍ | 576k/3.87M [00:00<00:01, 3.05MB/s]
.. parsed-literal::
24%|██▍ | 952k/3.87M [00:00<00:00, 3.36MB/s]
.. parsed-literal::
33%|███▎ | 1.30M/3.87M [00:00<00:00, 3.51MB/s]
.. parsed-literal::
43%|████▎ | 1.67M/3.87M [00:00<00:00, 3.60MB/s]
.. parsed-literal::
52%|█████▏ | 2.02M/3.87M [00:00<00:00, 3.16MB/s]
.. parsed-literal::
61%|██████ | 2.36M/3.87M [00:00<00:00, 3.26MB/s]
.. parsed-literal::
70%|███████ | 2.72M/3.87M [00:00<00:00, 3.38MB/s]
.. parsed-literal::
80%|███████▉ | 3.09M/3.87M [00:00<00:00, 3.49MB/s]
.. parsed-literal::
89%|████████▉ | 3.46M/3.87M [00:01<00:00, 3.58MB/s]
.. parsed-literal::
99%|█████████▉| 3.84M/3.87M [00:01<00:00, 3.67MB/s]
.. parsed-literal::
100%|██████████| 3.87M/3.87M [00:01<00:00, 3.45MB/s]
.. parsed-literal::
Fusing layers...
.. parsed-literal::
YOLOv5n summary: 213 layers, 1867405 parameters, 0 gradients
Adding AutoShape...
.. parsed-literal::
Adding AutoShape...
Hardware
@ -209,10 +288,10 @@ benchmarking process.
.. code:: ipython3
import openvino as ov
# initialize OpenVINO
core = ov.Core()
# print available devices
for device in core.available_devices:
device_name = core.get_property(device, "FULL_DEVICE_NAME")
@ -236,8 +315,8 @@ and prints two measures: seconds per image and frames per second (FPS).
.. code:: ipython3
INFER_NUMBER = 1000
def benchmark_model(model: Any, input_data: np.ndarray, benchmark_name: str, device_name: str = "CPU") -> float:
"""
Helper function for benchmarking the model. It measures the time and prints results.
@ -248,21 +327,21 @@ and prints two measures: seconds per image and frames per second (FPS).
end = time.perf_counter()
first_infer_time = end - start
print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")
# benchmarking
start = time.perf_counter()
for _ in range(INFER_NUMBER):
model(input_data)
end = time.perf_counter()
# elapsed time
infer_time = end - start
# print second per image and FPS
mean_infer_time = infer_time / INFER_NUMBER
mean_fps = INFER_NUMBER / infer_time
print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")
return mean_infer_time
The following functions aim to post-process results and draw boxes on
@ -281,21 +360,21 @@ the image.
"cell phone", "microwave", "oven", "oaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear",
"hair drier", "toothbrush"
]
# Colors for the classes above (Rainbow Color Map).
colors = cv2.applyColorMap(
src=np.arange(0, 255, 255 / len(classes), dtype=np.float32).astype(np.uint8),
colormap=cv2.COLORMAP_RAINBOW,
).squeeze()
def postprocess(detections: np.ndarray) -> List[Tuple]:
"""
Postprocess the raw results from the model.
"""
# candidates - probability > 0.25
detections = detections[detections[..., 4] > 0.25]
boxes = []
labels = []
scores = []
@ -309,22 +388,22 @@ the image.
)
labels.append(int(label))
scores.append(float(score))
# Apply non-maximum suppression to get rid of many overlapping entities.
# See https://paperswithcode.com/method/non-maximum-suppression
# This algorithm returns indices of objects to keep.
indices = cv2.dnn.NMSBoxes(
bboxes=boxes, scores=scores, score_threshold=0.25, nms_threshold=0.5
)
# If there are no boxes.
if len(indices) == 0:
return []
# Filter detected objects.
return [(labels[idx], scores[idx], boxes[idx]) for idx in indices.flatten()]
def draw_boxes(img: np.ndarray, boxes):
"""
Draw detected boxes on the image.
@ -336,7 +415,7 @@ the image.
x2 = box[0] + box[2]
y2 = box[1] + box[3]
cv2.rectangle(img=img, pt1=box[:2], pt2=(x2, y2), color=color, thickness=2)
# Draw a label name inside the box.
cv2.putText(
img=img,
@ -348,17 +427,17 @@ the image.
thickness=1,
lineType=cv2.LINE_AA,
)
def show_result(results: np.ndarray):
"""
Postprocess the raw results, draw boxes and show the image.
"""
output_img = image.copy()
detections = postprocess(results)
draw_boxes(output_img, detections)
utils.show_array(output_img)
Optimizations
@ -381,7 +460,7 @@ optimizations applied. We will treat it as our baseline.
.. code:: ipython3
import torch
with torch.no_grad():
result = pytorch_model(torch.as_tensor(input_image)).detach().numpy()[0]
show_result(result)
@ -394,8 +473,12 @@ optimizations applied. We will treat it as our baseline.
.. parsed-literal::
PyTorch model on CPU. First inference time: 0.0252 seconds
PyTorch model on CPU: 0.0214 seconds per image (46.73 FPS)
PyTorch model on CPU. First inference time: 0.0268 seconds
.. parsed-literal::
PyTorch model on CPU: 0.0213 seconds per image (46.98 FPS)
ONNX model
@ -411,12 +494,12 @@ Representation (IR) to leverage the OpenVINO Runtime.
.. code:: ipython3
onnx_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}").with_suffix(".onnx")
# export PyTorch model to ONNX if it doesn't already exist
if not onnx_path.exists():
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_path)
# load and compile in OpenVINO
onnx_model = core.read_model(onnx_path)
onnx_model = core.compile_model(onnx_model, device_name="CPU")
@ -435,7 +518,7 @@ Representation (IR) to leverage the OpenVINO Runtime.
result = onnx_model(input_image)[onnx_model.output(0)][0]
show_result(result)
onnx_infer_time = benchmark_model(model=onnx_model, input_data=input_image, benchmark_name="ONNX model")
del onnx_model # release resources
@ -446,7 +529,11 @@ Representation (IR) to leverage the OpenVINO Runtime.
.. parsed-literal::
ONNX model on CPU. First inference time: 0.0186 seconds
ONNX model on CPU: 0.0124 seconds per image (80.72 FPS)
.. parsed-literal::
ONNX model on CPU: 0.0123 seconds per image (81.52 FPS)
OpenVINO IR model
@ -467,13 +554,13 @@ accuracy drop. That’s why we skip that step in this notebook.
ov_model = ov.convert_model(onnx_path)
# save the model on disk
ov.save_model(ov_model, str(onnx_path.with_suffix(".xml")))
ov_cpu_model = core.compile_model(ov_model, device_name="CPU")
result = ov_cpu_model(input_image)[ov_cpu_model.output(0)][0]
show_result(result)
ov_cpu_infer_time = benchmark_model(model=ov_cpu_model, input_data=input_image, benchmark_name="OpenVINO model")
del ov_cpu_model # release resources
@ -483,8 +570,12 @@ accuracy drop. That’s why we skip that step in this notebook.
.. parsed-literal::
OpenVINO model on CPU. First inference time: 0.0148 seconds
OpenVINO model on CPU: 0.0123 seconds per image (81.38 FPS)
OpenVINO model on CPU. First inference time: 0.0166 seconds
.. parsed-literal::
OpenVINO model on CPU: 0.0122 seconds per image (81.86 FPS)
OpenVINO IR model on GPU
@ -506,11 +597,11 @@ execution.
ov_gpu_infer_time = 0.0
if "GPU" in core.available_devices:
ov_gpu_model = core.compile_model(ov_model, device_name="GPU")
result = ov_gpu_model(input_image)[ov_gpu_model.output(0)][0]
show_result(result)
ov_gpu_infer_time = benchmark_model(model=ov_gpu_model, input_data=input_image, benchmark_name="OpenVINO model", device_name="GPU")
del ov_gpu_model # release resources
OpenVINO IR model + more inference threads
@ -521,7 +612,7 @@ OpenVINO IR model + more inference threads
There is a possibility to add a config for any device (CPU in this
case). We will increase the number of threads to an equal number of our
cores. There are `more
options <https://docs.openvino.ai/2023.3/groupov_runtime_cpp_prop_api.html>`__
options <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ov__runtime__cpp__prop__api.html>`__
to be changed, so it’s worth playing with them to see what works best in
our case. In some cases, this optimization may worsen the performance.
If it is the case, don’t use it.
@ -529,13 +620,13 @@ If it is the case, don’t use it.
.. code:: ipython3
num_cores = os.cpu_count()
ov_cpu_config_model = core.compile_model(ov_model, device_name="CPU", config={"INFERENCE_NUM_THREADS": num_cores})
result = ov_cpu_config_model(input_image)[ov_cpu_config_model.output(0)][0]
show_result(result)
ov_cpu_config_infer_time = benchmark_model(model=ov_cpu_config_model, input_data=input_image, benchmark_name="OpenVINO model + more threads")
del ov_cpu_config_model # release resources
@ -545,8 +636,12 @@ If it is the case, don’t use it.
.. parsed-literal::
OpenVINO model + more threads on CPU. First inference time: 0.0155 seconds
OpenVINO model + more threads on CPU: 0.0124 seconds per image (80.47 FPS)
OpenVINO model + more threads on CPU. First inference time: 0.0153 seconds
.. parsed-literal::
OpenVINO model + more threads on CPU: 0.0122 seconds per image (82.19 FPS)
OpenVINO IR model in latency mode
@ -565,7 +660,7 @@ devices as well.
.. code:: ipython3
ov_auto_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "LATENCY"})
result = ov_auto_model(input_image)[ov_auto_model.output(0)][0]
show_result(result)
ov_auto_infer_time = benchmark_model(model=ov_auto_model, input_data=input_image, benchmark_name="OpenVINO model", device_name="AUTO")
@ -577,8 +672,12 @@ devices as well.
.. parsed-literal::
OpenVINO model on AUTO. First inference time: 0.0156 seconds
OpenVINO model on AUTO: 0.0125 seconds per image (79.73 FPS)
OpenVINO model on AUTO. First inference time: 0.0154 seconds
.. parsed-literal::
OpenVINO model on AUTO: 0.0125 seconds per image (80.26 FPS)
OpenVINO IR model in latency mode + shared memory
@ -599,11 +698,11 @@ performance!
# it must be assigned to a variable, not to be garbage collected
c_input_image = np.ascontiguousarray(input_image, dtype=np.float32)
input_tensor = ov.Tensor(c_input_image, shared_memory=True)
result = ov_auto_model(input_tensor)[ov_auto_model.output(0)][0]
show_result(result)
ov_auto_shared_infer_time = benchmark_model(model=ov_auto_model, input_data=input_tensor, benchmark_name="OpenVINO model + shared memory", device_name="AUTO")
del ov_auto_model # release resources
@ -613,8 +712,12 @@ performance!
.. parsed-literal::
OpenVINO model + shared memory on AUTO. First inference time: 0.0112 seconds
OpenVINO model + shared memory on AUTO: 0.0054 seconds per image (185.55 FPS)
OpenVINO model + shared memory on AUTO. First inference time: 0.0124 seconds
.. parsed-literal::
OpenVINO model + shared memory on AUTO: 0.0054 seconds per image (184.61 FPS)
Other tricks
@ -625,9 +728,9 @@ Other tricks
There are other tricks for performance improvement, such as quantization
and pre-post-processing or dedicated to throughput mode. To get even
more from your model, please visit
`111-detection-quantization <../111-detection-quantization>`__,
`118-optimize-preprocessing <../118-optimize-preprocessing>`__, and
`109-throughput-tricks <109-throughput-tricks.ipynb>`__.
`111-detection-quantization <111-yolov5-quantization-migration-with-output.html>`__,
`118-optimize-preprocessing <118-optimize-preprocessing-with-output.html>`__, and
`109-throughput-tricks <109-throughput-tricks-with-output.html>`__.
Performance comparison
----------------------
@ -645,21 +748,21 @@ steps, just skip them.
.. code:: ipython3
from matplotlib import pyplot as plt
labels = ["PyTorch model", "ONNX model", "OpenVINO IR model", "OpenVINO IR model on GPU", "OpenVINO IR model + more inference threads",
"OpenVINO IR model in latency mode", "OpenVINO IR model in latency mode + shared memory"]
# make them milliseconds
times = list(map(lambda x: 1000 * x, [pytorch_infer_time, onnx_infer_time, ov_cpu_infer_time, ov_gpu_infer_time, ov_cpu_config_infer_time,
ov_auto_infer_time, ov_auto_shared_infer_time]))
bar_colors = colors[::10] / 255.0
fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(labels, times, color=bar_colors)
ax.set_ylabel("Inference time [ms]")
ax.set_title("Performance difference")
plt.xticks(rotation='vertical')
plt.show()

4
docs/notebooks/109-latency-tricks-with-output_files/109-latency-tricks-with-output_31_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:545f14f9a5a612fe9e498660b4640562ad87b9b29df063fa6b354df83c16eaa8
size 57156
oid sha256:4f157b7f097e536fab10e5a103ae836be062794d8d7b4f9b16185e2e6900d2f5
size 56983

20
docs/notebooks/109-latency-tricks-with-output_files/index.html → docs/notebooks/109-latency-tricks-with-output_files/index.html.tmp
View File

@ -1,14 +1,14 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/109-latency-tricks-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/109-latency-tricks-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/109-latency-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-latency-tricks-with-output_15_0.jpg">109-latency-tricks-with-output_15_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_18_0.jpg">109-latency-tricks-with-output_18_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_20_0.jpg">109-latency-tricks-with-output_20_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_24_0.jpg">109-latency-tricks-with-output_24_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_26_0.jpg">109-latency-tricks-with-output_26_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_28_0.jpg">109-latency-tricks-with-output_28_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-latency-tricks-with-output_31_0.png">109-latency-tricks-with-output_31_0.png</a> 07-Dec-2023 00:49 57156
<a href="109-latency-tricks-with-output_5_0.jpg">109-latency-tricks-with-output_5_0.jpg</a> 07-Dec-2023 00:49 155828
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/109-latency-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-latency-tricks-with-output_15_0.jpg">109-latency-tricks-with-output_15_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_18_0.jpg">109-latency-tricks-with-output_18_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_20_0.jpg">109-latency-tricks-with-output_20_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_24_0.jpg">109-latency-tricks-with-output_24_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_26_0.jpg">109-latency-tricks-with-output_26_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_28_0.jpg">109-latency-tricks-with-output_28_0.jpg</a> 26-Jan-2024 01:04 162715
<a href="109-latency-tricks-with-output_31_0.png">109-latency-tricks-with-output_31_0.png</a> 26-Jan-2024 01:04 56983
<a href="109-latency-tricks-with-output_5_0.jpg">109-latency-tricks-with-output_5_0.jpg</a> 26-Jan-2024 01:04 155828
</pre><hr></body>
</html>

210
docs/notebooks/109-throughput-tricks-with-output.rst
View File

@ -24,8 +24,8 @@ The quantization and pre-post-processing API are not included here as
they change the precision (quantization) or processing graph
(prepostprocessor). You can find examples of how to apply them to
optimize performance on OpenVINO IR files in
`111-detection-quantization <../111-detection-quantization>`__ and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
`111-detection-quantization <111-yolov5-quantization-migration-with-output.html>`__ and
`118-optimize-preprocessing <118-optimize-preprocessing-with-output.html>`__.
|image0|
@ -41,10 +41,10 @@ optimize performance on OpenVINO IR files in
result in different performance.
A similar notebook focused on the latency mode is available
`here <109-latency-tricks.ipynb>`__.
**Table of contents:**
`here <109-latency-tricks-with-output.html>`__.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Data <#data>`__
@ -93,7 +93,7 @@ Prerequisites
import time
from pathlib import Path
from typing import Any, List, Tuple
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -116,24 +116,24 @@ object detection model.
import numpy as np
import cv2
FRAMES_NUMBER = 1024
IMAGE_WIDTH = 640
IMAGE_HEIGHT = 480
# load image
image = utils.load_image("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg")
image = cv2.resize(image, dsize=(IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_AREA)
# preprocess it for YOLOv5
input_image = image / 255.0
input_image = np.transpose(input_image, axes=(2, 0, 1))
input_image = np.expand_dims(input_image, axis=0)
# simulate video with many frames
video_frames = np.tile(input_image, (FRAMES_NUMBER, 1, 1, 1, 1))
# show the image
utils.show_array(image)
@ -146,7 +146,7 @@ object detection model.
.. parsed-literal::
<DisplayHandle display_id=ffb43bd5831f251ba3c21b52bdd4c3fc>
<DisplayHandle display_id=27f64c41c38d04ac2269516dbbad4c95>
@ -164,13 +164,13 @@ PyTorch Hub and small enough to see the difference in performance.
import torch
from IPython.utils import io
# directory for all models
base_model_dir = Path("model")
model_name = "yolov5n"
model_path = base_model_dir / model_name
# load YOLOv5n from PyTorch Hub
pytorch_model = torch.hub.load("ultralytics/yolov5", "custom", path=model_path, device="cpu", skip_validation=True)
# don't print full model architecture
@ -181,11 +181,27 @@ PyTorch Hub and small enough to see the difference in performance.
.. parsed-literal::
Using cache found in /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master
.. parsed-literal::
YOLOv5 🚀 2023-4-21 Python-3.8.10 torch-2.1.0+cpu CPU
Fusing layers...
.. parsed-literal::
Fusing layers...
.. parsed-literal::
YOLOv5n summary: 213 layers, 1867405 parameters, 0 gradients
Adding AutoShape...
.. parsed-literal::
Adding AutoShape...
.. parsed-literal::
@ -207,10 +223,10 @@ benchmarking process.
.. code:: ipython3
import openvino as ov
# initialize OpenVINO
core = ov.Core()
# print available devices
for device in core.available_devices:
device_name = core.get_property(device, "FULL_DEVICE_NAME")
@ -234,8 +250,8 @@ second (FPS).
.. code:: ipython3
from openvino.runtime import AsyncInferQueue
def benchmark_model(model: Any, frames: np.ndarray, async_queue: AsyncInferQueue = None, benchmark_name: str = "OpenVINO model", device_name: str = "CPU") -> float:
"""
Helper function for benchmarking the model. It measures the time and prints results.
@ -248,7 +264,7 @@ second (FPS).
end = time.perf_counter()
first_infer_time = end - start
print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")
# benchmarking
start = time.perf_counter()
for batch in frames:
@ -257,15 +273,15 @@ second (FPS).
if async_queue:
async_queue.wait_all()
end = time.perf_counter()
# elapsed time
infer_time = end - start
# print second per image and FPS
mean_infer_time = infer_time / FRAMES_NUMBER
mean_fps = FRAMES_NUMBER / infer_time
print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")
return mean_fps
The following functions aim to post-process results and draw boxes on
@ -284,21 +300,21 @@ the image.
"cell phone", "microwave", "oven", "oaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear",
"hair drier", "toothbrush"
]
# Colors for the classes above (Rainbow Color Map).
colors = cv2.applyColorMap(
src=np.arange(0, 255, 255 / len(classes), dtype=np.float32).astype(np.uint8),
colormap=cv2.COLORMAP_RAINBOW,
).squeeze()
def postprocess(detections: np.ndarray) -> List[Tuple]:
"""
Postprocess the raw results from the model.
"""
# candidates - probability > 0.25
detections = detections[detections[..., 4] > 0.25]
boxes = []
labels = []
scores = []
@ -312,22 +328,22 @@ the image.
)
labels.append(int(label))
scores.append(float(score))
# Apply non-maximum suppression to get rid of many overlapping entities.
# See https://paperswithcode.com/method/non-maximum-suppression
# This algorithm returns indices of objects to keep.
indices = cv2.dnn.NMSBoxes(
bboxes=boxes, scores=scores, score_threshold=0.25, nms_threshold=0.5
)
# If there are no boxes.
if len(indices) == 0:
return []
# Filter detected objects.
return [(labels[idx], scores[idx], boxes[idx]) for idx in indices.flatten()]
def draw_boxes(img: np.ndarray, boxes):
"""
Draw detected boxes on the image.
@ -339,7 +355,7 @@ the image.
x2 = box[0] + box[2]
y2 = box[1] + box[3]
cv2.rectangle(img=img, pt1=box[:2], pt2=(x2, y2), color=color, thickness=2)
# Draw a label name inside the box.
cv2.putText(
img=img,
@ -351,17 +367,17 @@ the image.
thickness=1,
lineType=cv2.LINE_AA,
)
def show_result(results: np.ndarray):
"""
Postprocess the raw results, draw boxes and show the image.
"""
output_img = image.copy()
detections = postprocess(results)
draw_boxes(output_img, detections)
utils.show_array(output_img)
Optimizations
@ -384,7 +400,7 @@ optimizations applied. We will treat it as our baseline.
.. code:: ipython3
import torch
with torch.no_grad():
result = pytorch_model(torch.as_tensor(video_frames[0])).detach().numpy()[0]
show_result(result)
@ -397,8 +413,12 @@ optimizations applied. We will treat it as our baseline.
.. parsed-literal::
PyTorch model on CPU. First inference time: 0.0220 seconds
PyTorch model on CPU: 0.0208 seconds per image (48.18 FPS)
PyTorch model on CPU. First inference time: 0.0201 seconds
.. parsed-literal::
PyTorch model on CPU: 0.0192 seconds per image (52.19 FPS)
OpenVINO IR model
@ -418,23 +438,23 @@ step in this notebook.
.. code:: ipython3
onnx_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}").with_suffix(".onnx")
# export PyTorch model to ONNX if it doesn't already exist
if not onnx_path.exists():
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_path)
# convert ONNX model to IR, use FP16
ov_model = ov.convert_model(onnx_path)
.. code:: ipython3
ov_cpu_model = core.compile_model(ov_model, device_name="CPU")
result = ov_cpu_model(video_frames[0])[ov_cpu_model.output(0)][0]
show_result(result)
ov_cpu_fps = benchmark_model(model=ov_cpu_model, frames=video_frames, benchmark_name="OpenVINO model")
del ov_cpu_model # release resources
@ -444,8 +464,12 @@ step in this notebook.
.. parsed-literal::
OpenVINO model on CPU. First inference time: 0.0156 seconds
OpenVINO model on CPU: 0.0071 seconds per image (141.19 FPS)
OpenVINO model on CPU. First inference time: 0.0134 seconds
.. parsed-literal::
OpenVINO model on CPU: 0.0070 seconds per image (142.35 FPS)
OpenVINO IR model + bigger batch
@ -464,13 +488,13 @@ hardware and model.
.. code:: ipython3
batch_size = 4
onnx_batch_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}_batch_{batch_size}").with_suffix(".onnx")
if not onnx_batch_path.exists():
dummy_input = torch.randn(batch_size, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_batch_path)
# export the model with the bigger batch size
ov_batch_model = ov.convert_model(onnx_batch_path)
@ -486,13 +510,13 @@ hardware and model.
.. code:: ipython3
ov_cpu_batch_model = core.compile_model(ov_batch_model, device_name="CPU")
batched_video_frames = video_frames.reshape([-1, batch_size, 3, IMAGE_HEIGHT, IMAGE_WIDTH])
result = ov_cpu_batch_model(batched_video_frames[0])[ov_cpu_batch_model.output(0)][0]
show_result(result)
ov_cpu_batch_fps = benchmark_model(model=ov_cpu_batch_model, frames=batched_video_frames, benchmark_name="OpenVINO model + bigger batch")
del ov_cpu_batch_model # release resources
@ -502,8 +526,12 @@ hardware and model.
.. parsed-literal::
OpenVINO model + bigger batch on CPU. First inference time: 0.0481 seconds
OpenVINO model + bigger batch on CPU: 0.0069 seconds per image (145.67 FPS)
OpenVINO model + bigger batch on CPU. First inference time: 0.0435 seconds
.. parsed-literal::
OpenVINO model + bigger batch on CPU: 0.0068 seconds per image (147.59 FPS)
Asynchronous processing
@ -533,17 +561,17 @@ the pipeline.
result = infer_request.get_output_tensor(0).data[0]
show_result(result)
pass
infer_queue = ov.AsyncInferQueue(ov_model)
infer_queue.set_callback(callback) # set callback to post-process (show) results
infer_queue.start_async(video_frames[0])
infer_queue.wait_all()
# don't show output for the remaining frames
infer_queue.set_callback(lambda x, y: {})
fps = benchmark_model(model=infer_queue.start_async, frames=video_frames, async_queue=infer_queue, benchmark_name=benchmark_name, device_name=device_name)
del infer_queue # release resources
return fps
@ -564,9 +592,9 @@ feature, which sets the batch size to the optimal level.
.. code:: ipython3
ov_cpu_through_model = core.compile_model(ov_model, device_name="CPU", config={"PERFORMANCE_HINT": "THROUGHPUT"})
ov_cpu_through_fps = benchmark_async_mode(ov_cpu_through_model, benchmark_name="OpenVINO model", device_name="CPU (THROUGHPUT)")
del ov_cpu_through_model # release resources
@ -576,8 +604,12 @@ feature, which sets the batch size to the optimal level.
.. parsed-literal::
OpenVINO model on CPU (THROUGHPUT). First inference time: 0.0254 seconds
OpenVINO model on CPU (THROUGHPUT): 0.0040 seconds per image (250.82 FPS)
OpenVINO model on CPU (THROUGHPUT). First inference time: 0.0260 seconds
.. parsed-literal::
OpenVINO model on CPU (THROUGHPUT): 0.0040 seconds per image (248.03 FPS)
OpenVINO IR model in throughput mode on GPU
@ -601,9 +633,9 @@ execution.
if "GPU" in core.available_devices:
# compile for GPU
ov_gpu_model = core.compile_model(ov_model, device_name="GPU", config={"PERFORMANCE_HINT": "THROUGHPUT"})
ov_gpu_fps = benchmark_async_mode(ov_gpu_model, benchmark_name="OpenVINO model", device_name="GPU (THROUGHPUT)")
del ov_gpu_model # release resources
OpenVINO IR model in throughput mode on AUTO
@ -619,9 +651,9 @@ performance hint.
.. code:: ipython3
ov_auto_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "THROUGHPUT"})
ov_auto_fps = benchmark_async_mode(ov_auto_model, benchmark_name="OpenVINO model", device_name="AUTO (THROUGHPUT)")
del ov_auto_model # release resources
@ -631,8 +663,12 @@ performance hint.
.. parsed-literal::
OpenVINO model on AUTO (THROUGHPUT). First inference time: 0.0231 seconds
OpenVINO model on AUTO (THROUGHPUT): 0.0040 seconds per image (251.86 FPS)
OpenVINO model on AUTO (THROUGHPUT). First inference time: 0.0244 seconds
.. parsed-literal::
OpenVINO model on AUTO (THROUGHPUT): 0.0040 seconds per image (250.62 FPS)
OpenVINO IR model in cumulative throughput mode on AUTO
@ -649,7 +685,7 @@ activate all devices.
.. code:: ipython3
ov_auto_cumulative_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "CUMULATIVE_THROUGHPUT"})
ov_auto_cumulative_fps = benchmark_async_mode(ov_auto_cumulative_model, benchmark_name="OpenVINO model", device_name="AUTO (CUMULATIVE THROUGHPUT)")
@ -659,8 +695,12 @@ activate all devices.
.. parsed-literal::
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT). First inference time: 0.0260 seconds
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT): 0.0040 seconds per image (251.22 FPS)
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT). First inference time: 0.0211 seconds
.. parsed-literal::
OpenVINO model on AUTO (CUMULATIVE THROUGHPUT): 0.0040 seconds per image (249.40 FPS)
Other tricks
@ -673,9 +713,9 @@ options, quantization and pre-post-processing or dedicated to latency
mode. To get even more from your model, please visit `advanced
throughput
options <https://docs.openvino.ai/2023.3/openvino_docs_deployment_optimization_guide_tput_advanced.html>`__,
`109-latency-tricks <109-latency-tricks.ipynb>`__,
`111-detection-quantization <../111-detection-quantization>`__, and
`118-optimize-preprocessing <../118-optimize-preprocessing>`__.
`109-latency-tricks <109-latency-tricks-with-output.html-with-output.html>`__,
`111-detection-quantization <111-yolov5-quantization-migration-with-output.html>`__, and
`118-optimize-preprocessing <118-optimize-preprocessing-with-output.html>`__.
Performance comparison
----------------------
@ -693,20 +733,20 @@ steps, just skip them.
.. code:: ipython3
from matplotlib import pyplot as plt
labels = ["PyTorch model", "OpenVINO IR model", "OpenVINO IR model + bigger batch", "OpenVINO IR model in throughput mode", "OpenVINO IR model in throughput mode on GPU",
"OpenVINO IR model in throughput mode on AUTO", "OpenVINO IR model in cumulative throughput mode on AUTO"]
fps = [pytorch_fps, ov_cpu_fps, ov_cpu_batch_fps, ov_cpu_through_fps, ov_gpu_fps, ov_auto_fps, ov_auto_cumulative_fps]
bar_colors = colors[::10] / 255.0
fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(labels, fps, color=bar_colors)
ax.set_ylabel("Throughput [FPS]")
ax.set_title("Performance difference")
plt.xticks(rotation='vertical')
plt.show()
@ -725,6 +765,6 @@ object detection model. Even if you experience much better performance
after running this notebook, please note this may not be valid for every
hardware or every model. For the most accurate results, please use
``benchmark_app`` `command-line
tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html>`__.
tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool>`__.
Note that ``benchmark_app`` cannot measure the impact of some tricks
above.

4
docs/notebooks/109-throughput-tricks-with-output_files/109-throughput-tricks-with-output_34_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b94c59d38427a1ec4994131e2bbefc03b0ebdf0fde6f20956a3775faa4e5f451
size 62516
oid sha256:8e1932b29ece3712ada3af1ea8df832692a5416111c7caeb3c884cd3ee628110
size 62463

20
docs/notebooks/109-throughput-tricks-with-output_files/index.html → docs/notebooks/109-throughput-tricks-with-output_files/index.html.tmp
View File

@ -1,14 +1,14 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/109-throughput-tricks-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/109-throughput-tricks-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/109-throughput-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-throughput-tricks-with-output_15_0.jpg">109-throughput-tricks-with-output_15_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_18_0.jpg">109-throughput-tricks-with-output_18_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_21_0.jpg">109-throughput-tricks-with-output_21_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_25_0.jpg">109-throughput-tricks-with-output_25_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_29_0.jpg">109-throughput-tricks-with-output_29_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_31_0.jpg">109-throughput-tricks-with-output_31_0.jpg</a> 07-Dec-2023 00:49 162715
<a href="109-throughput-tricks-with-output_34_0.png">109-throughput-tricks-with-output_34_0.png</a> 07-Dec-2023 00:49 62516
<a href="109-throughput-tricks-with-output_5_0.jpg">109-throughput-tricks-with-output_5_0.jpg</a> 07-Dec-2023 00:49 155828
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/109-throughput-tricks-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="109-throughput-tricks-with-output_15_0.jpg">109-throughput-tricks-with-output_15_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_18_0.jpg">109-throughput-tricks-with-output_18_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_21_0.jpg">109-throughput-tricks-with-output_21_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_25_0.jpg">109-throughput-tricks-with-output_25_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_29_0.jpg">109-throughput-tricks-with-output_29_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_31_0.jpg">109-throughput-tricks-with-output_31_0.jpg</a> 26-Jan-2024 01:05 162715
<a href="109-throughput-tricks-with-output_34_0.png">109-throughput-tricks-with-output_34_0.png</a> 26-Jan-2024 01:05 62463
<a href="109-throughput-tricks-with-output_5_0.jpg">109-throughput-tricks-with-output_5_0.jpg</a> 26-Jan-2024 01:05 155828
</pre><hr></body>
</html>

188
docs/notebooks/110-ct-scan-live-inference-with-output.rst
View File

@ -29,13 +29,12 @@ notebook.
For demonstration purposes, this tutorial will download one converted CT
scan to use for inference.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Benchmark Model
Performance <#benchmark-model-performance>`__
- `Benchmark Model Performance <#benchmark-model-performance>`__
- `Download and Prepare Data <#download-and-prepare-data>`__
- `Show Live Inference <#show-live-inference>`__
@ -57,8 +56,10 @@ scan to use for inference.
Note: you may need to restart the kernel to use updated packages.
Imports
-------------------------------------------------
Imports
-------
.. code:: ipython3
@ -66,27 +67,33 @@ Imports
import sys
import zipfile
from pathlib import Path
import numpy as np
from monai.transforms import LoadImage
import openvino as ov
from custom_segmentation import SegmentationModel
sys.path.append("../utils")
from notebook_utils import download_file
.. parsed-literal::
2023-10-30 22:42:33.368243: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-10-30 22:42:33.402770: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 22:50:13.572016: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 22:50:13.606068: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-30 22:42:34.097093: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Settings
--------------------------------------------------
.. parsed-literal::
2024-01-25 22:50:14.166493: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Settings
--------
To use the pre-trained models, set ``IR_PATH`` to
``"pretrained_model/unet44.xml"`` and ``COMPRESSED_MODEL_PATH`` to
@ -97,16 +104,16 @@ trained or optimized yourself, adjust the model paths.
# The directory that contains the IR model (xml and bin) files.
models_dir = Path('pretrained_model')
ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/FP16-INT8/'
ir_model_name_xml = 'quantized_unet_kits19.xml'
ir_model_name_bin = 'quantized_unet_kits19.bin'
download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory=models_dir)
download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory=models_dir)
MODEL_PATH = models_dir / ir_model_name_xml
# Uncomment the next line to use the FP16 model instead of the quantized model.
# MODEL_PATH = "pretrained_model/unet_kits19.xml"
@ -123,10 +130,11 @@ trained or optimized yourself, adjust the model paths.
pretrained_model/quantized_unet_kits19.bin: 0%| | 0.00/1.90M [00:00<?, ?B/s]
Benchmark Model Performance
---------------------------------------------------------------------
Benchmark Model Performance
---------------------------
To measure the inference performance of the IR model, use `Benchmark
To measure the inference
performance of the IR model, use `Benchmark
Tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html>`__
- an inference performance measurement tool in OpenVINO. Benchmark tool
is a command-line application that can be run in the notebook with
@ -146,16 +154,16 @@ is a command-line application that can be run in the notebook with
core = ov.Core()
# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.
device_list = ["MULTI:CPU,GPU" if "GPU" in core.available_devices else "AUTO"]
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + device_list,
value=device_list[0],
description='Device:',
disabled=False,
)
device
@ -179,18 +187,18 @@ is a command-line application that can be run in the notebook with
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.1.0-12185-9e6b00e51cd-releases/2023/1
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2023.1.0-12185-9e6b00e51cd-releases/2023/1
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 14.42 ms
[ INFO ] Read model took 13.17 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input.1 (node: input.1) : f32 / [...] / [1,1,512,512]
@ -204,7 +212,11 @@ is a command-line application that can be run in the notebook with
[ INFO ] Model outputs:
[ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 219.65 ms
.. parsed-literal::
[ INFO ] Compile model took 235.31 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: pretrained_unet_kits19
@ -225,32 +237,42 @@ is a command-line application that can be run in the notebook with
[ INFO ] NETWORK_NAME: pretrained_unet_kits19
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] PERF_COUNT: False
[ INFO ] PERF_COUNT: NO
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input.1'!. This input will be filled with random values!
[ INFO ] Fill input 'input.1' with random values
[ INFO ] Fill input 'input.1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 29.06 ms
.. parsed-literal::
[ INFO ] First inference took 24.14 ms
.. parsed-literal::
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1347 iterations
[ INFO ] Duration: 15006.65 ms
[ INFO ] Count: 1355 iterations
[ INFO ] Duration: 15007.81 ms
[ INFO ] Latency:
[ INFO ] Median: 10.90 ms
[ INFO ] Average: 10.96 ms
[ INFO ] Min: 10.61 ms
[ INFO ] Max: 14.44 ms
[ INFO ] Throughput: 89.76 FPS
[ INFO ] Median: 10.84 ms
[ INFO ] Average: 10.89 ms
[ INFO ] Min: 10.58 ms
[ INFO ] Max: 14.29 ms
[ INFO ] Throughput: 90.29 FPS
Download and Prepare Data
-------------------------
Download and Prepare Data
-------------------------------------------------------------------
Download one validation video for live inference.
@ -270,9 +292,9 @@ downloaded and extracted in the next cell.
# The CT scan case number. For example: 16 for data from the case_00016 directory.
# Currently only 117 is supported.
CASE = 117
case_path = BASEDIR / f"case_{CASE:05d}"
if not case_path.exists():
filename = download_file(
f"https://storage.openvinotoolkit.org/data/test_data/openvino_notebooks/kits19/case_{CASE:05d}.zip"
@ -296,8 +318,10 @@ downloaded and extracted in the next cell.
Downloaded and extracted data for case_00117
Show Live Inference
-------------------------------------------------------------
Show Live Inference
-------------------
To show live inference on the model in the notebook, use the
asynchronous processing feature of OpenVINO Runtime.
@ -327,8 +351,10 @@ inference queue, there are two jobs to do:
Everything else will be handled by the ``AsyncInferQueue`` instance.
Load Model and List of Image Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load Model and List of Image Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the segmentation model to OpenVINO Runtime with
``SegmentationModel``, based on the Model API from `Open Model
@ -345,7 +371,7 @@ to see the implementation.
ie=core, model_path=Path(MODEL_PATH), sigmoid=True, rotate_and_flip=True
)
image_paths = sorted(case_path.glob("imaging_frames/*jpg"))
print(f"{case_path.name}, {len(image_paths)} images")
@ -354,8 +380,10 @@ to see the implementation.
case_00117, 69 images
Prepare images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prepare images
~~~~~~~~~~~~~~
Use the ``reader = LoadImage()`` function to read the images in the same
way as in the
@ -365,18 +393,20 @@ tutorial.
.. code:: ipython3
framebuf = []
next_frame_id = 0
reader = LoadImage(image_only=True, dtype=np.uint8)
while next_frame_id < len(image_paths) - 1:
image_path = image_paths[next_frame_id]
image = reader(str(image_path))
framebuf.append(image)
next_frame_id += 1
Specify device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Specify device
~~~~~~~~~~~~~~
.. code:: ipython3
@ -391,8 +421,10 @@ Specify device
Setting callback function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Setting callback function
~~~~~~~~~~~~~~~~~~~~~~~~~
When ``callback`` is set, any job that ends the inference, calls the
Python function. The ``callback`` function must have two arguments: one
@ -407,58 +439,60 @@ The ``callback`` function will show the results of inference.
import cv2
import copy
from IPython import display
from typing import Dict, Any
# Define a callback function that runs every time the asynchronous pipeline completes inference on a frame
def completion_callback(infer_request: ov.InferRequest, user_data: Dict[str, Any],) -> None:
preprocess_meta = user_data['preprocess_meta']
raw_outputs = {out.any_name: copy.deepcopy(res.data) for out, res in zip(infer_request.model_outputs, infer_request.output_tensors)}
frame = segmentation_model.postprocess(raw_outputs, preprocess_meta)
_, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
# Create IPython image
i = display.Image(data=encoded_img)
# Display the image in this notebook
display.clear_output(wait=True)
display.display(i)
Create asynchronous inference queue and perform it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create asynchronous inference queue and perform it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
import time
load_start_time = time.perf_counter()
compiled_model = core.compile_model(segmentation_model.net, device.value)
# Create asynchronous inference queue with optimal number of infer requests
infer_queue = ov.AsyncInferQueue(compiled_model)
infer_queue.set_callback(completion_callback)
load_end_time = time.perf_counter()
results = [None] * len(framebuf)
frame_number = 0
# Perform inference on every frame in the framebuffer
start_time = time.time()
for i, input_frame in enumerate(framebuf):
inputs, preprocessing_meta = segmentation_model.preprocess({segmentation_model.net.input(0): input_frame})
infer_queue.start_async(inputs, {'preprocess_meta': preprocessing_meta})
# Wait until all inference requests in the AsyncInferQueue are completed
infer_queue.wait_all()
stop_time = time.time()
# Calculate total inference time and FPS
total_time = stop_time - start_time
fps = len(framebuf) / total_time
time_per_frame = 1 / fps
time_per_frame = 1 / fps
print(f"Loaded model to {device} in {load_end_time-load_start_time:.2f} seconds.")
print(f'Total time to infer all frames: {total_time:.3f}s')
print(f'Time per frame: {time_per_frame:.6f}s ({fps:.3f} FPS)')
@ -469,7 +503,7 @@ Create asynchronous inference queue and perform it
.. parsed-literal::
Loaded model to Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') in 0.22 seconds.
Total time to infer all frames: 3.558s
Time per frame: 0.052326s (19.111 FPS)
Loaded model to Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') in 0.24 seconds.
Total time to infer all frames: 2.762s
Time per frame: 0.040619s (24.619 FPS)

7
docs/notebooks/110-ct-scan-live-inference-with-output_files/index.html
View File

@ -1,7 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/110-ct-scan-live-inference-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/110-ct-scan-live-inference-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-scan-live-inference-with-output_21_0.png">110-ct-scan-live-inference-with-output_21_0.png</a> 31-Oct-2023 00:35 48780
</pre><hr></body>
</html>

7
docs/notebooks/110-ct-scan-live-inference-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/110-ct-scan-live-inference-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/110-ct-scan-live-inference-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-scan-live-inference-with-output_21_0.png">110-ct-scan-live-inference-with-output_21_0.png</a> 26-Jan-2024 01:05 48780
</pre><hr></body>
</html>

261
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output.rst
View File

@ -24,13 +24,13 @@ This third tutorial in the series shows how to:
All notebooks in this series:
- `Data Preparation for 2D Segmentation of 3D Medical
Data <data-preparation-ct-scan.ipynb>`__
Data <data-preparation-ct-scan-with-output.html>`__
- `Train a 2D-UNet Medical Imaging Model with PyTorch
Lightning <pytorch-monai-training.ipynb>`__
Lightning <pytorch-monai-training-with-output.html>`__
- Convert and Quantize a Segmentation Model and Show Live Inference
(this notebook)
- `Live Inference and Benchmark CT-scan
data <110-ct-scan-live-inference.ipynb>`__
data <110-ct-scan-live-inference-with-output.html>`__
Instructions
------------
@ -39,7 +39,7 @@ This notebook needs a trained UNet model. We provide a pre-trained
model, trained for 20 epochs with the full
`Kits-19 <https://github.com/neheller/kits19>`__ frames dataset, which
has an F1 score on the validation set of 0.9. The training code is
available in `this notebook <pytorch-monai-training.ipynb>`__.
available in `this notebook <pytorch-monai-training-with-output.html>`__.
NNCF for PyTorch models requires a C++ compiler. On Windows, install
`Microsoft Visual Studio
@ -53,8 +53,8 @@ demonstration purposes, this tutorial will download one converted CT
scan and use that scan for quantization and inference. For production
purposes, use a representative dataset for quantizing the model.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Settings <#settings>`__
@ -66,14 +66,14 @@ purposes, use a representative dataset for quantizing the model.
- `Metric <#metric>`__
- `Quantization <#quantization>`__
- `Compare FP32 and INT8 Model <#compare-fp-and-int-model>`__
- `Compare FP32 and INT8 Model <#compare-fp32-and-int8-model>`__
- `Compare File Size <#compare-file-size>`__
- `Compare Metrics for the original model and the quantized model to
be sure that there no
degradation. <#compare-metrics-for-the-original-model-and-the-quantized-model-to-be-sure-that-there-no-degradation>`__
degradation. <#compare-metrics-for-the-original-model-and-the-quantized-model-to-be-sure-that-there-no-degradation->`__
- `Compare Performance of the FP32 IR Model and Quantized
Models <#compare-performance-of-the-fp-ir-model-and-quantized-models>`__
Models <#compare-performance-of-the-fp32-ir-model-and-quantized-models>`__
- `Visually Compare Inference
Results <#visually-compare-inference-results>`__
@ -106,28 +106,28 @@ Imports
# to find the required C++ tools. This code assumes that Visual Studio is installed in the default
# directory. If you have a different C++ compiler, please add the correct path to os.environ["PATH"]
# directly. Note that the C++ Redistributable is not enough to run this notebook.
# Adding the path to os.environ["LIB"] is not always required - it depends on the system's configuration
import sys
if sys.platform == "win32":
import distutils.command.build_ext
import os
from pathlib import Path
if sys.getwindowsversion().build >= 20000: # Windows 11
search_path = "**/Hostx64/x64/cl.exe"
else:
search_path = "**/Hostx86/x64/cl.exe"
VS_INSTALL_DIR_2019 = r"C:/Program Files (x86)/Microsoft Visual Studio"
VS_INSTALL_DIR_2022 = r"C:/Program Files/Microsoft Visual Studio"
cl_paths_2019 = sorted(list(Path(VS_INSTALL_DIR_2019).glob(search_path)))
cl_paths_2022 = sorted(list(Path(VS_INSTALL_DIR_2022).glob(search_path)))
cl_paths = cl_paths_2019 + cl_paths_2022
if len(cl_paths) == 0:
raise ValueError(
"Cannot find Visual Studio. This notebook requires an x64 C++ compiler. If you installed "
@ -157,9 +157,9 @@ Imports
import zipfile
from pathlib import Path
from typing import Union
warnings.filterwarnings("ignore", category=UserWarning)
import cv2
import matplotlib.pyplot as plt
import monai
@ -170,22 +170,26 @@ Imports
from monai.transforms import LoadImage
from nncf.common.logging.logger import set_log_level
from torchmetrics import F1Score as F1
set_log_level(logging.ERROR) # Disables all NNCF info and warning messages
from custom_segmentation import SegmentationModel
from async_pipeline import show_live_inference
sys.path.append("../utils")
from notebook_utils import download_file
.. parsed-literal::
2023-12-06 22:47:51.629108: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-06 22:47:51.662883: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 22:50:47.254337: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 22:50:47.287377: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-06 22:47:52.221639: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 22:50:47.852256: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -201,7 +205,7 @@ Settings
By default, this notebook will download one CT scan from the KITS19
dataset that will be used for quantization. To use the full dataset, set
``BASEDIR`` to the path of the dataset, as prepared according to the
`Data Preparation <data-preparation-ct-scan.ipynb>`__ notebook.
`Data Preparation <data-preparation-ct-scan-with-output.html>`__ notebook.
.. code:: ipython3
@ -222,20 +226,20 @@ notebook is a
`BasicUNet <https://docs.monai.io/en/stable/networks.html#basicunet>`__
model from `MONAI <https://monai.io>`__. We provide a pre-trained
checkpoint. To see how this model performs, check out the `training
notebook <pytorch-monai-training.ipynb>`__.
notebook <pytorch-monai-training-with-output.html>`__.
.. code:: ipython3
state_dict_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/unet_kits19_state_dict.pth"
state_dict_file = download_file(state_dict_url, directory="pretrained_model")
state_dict = torch.load(state_dict_file, map_location=torch.device("cpu"))
new_state_dict = {}
for k, v in state_dict.items():
new_key = k.replace("_model.", "")
new_state_dict[new_key] = v
new_state_dict.pop("loss_function.pos_weight")
model = monai.networks.nets.BasicUNet(spatial_dims=2, in_channels=1, out_channels=1).eval()
model.load_state_dict(new_state_dict)
@ -300,7 +304,7 @@ Dataset
The ``KitsDataset`` class in the next cell expects images and masks in
the *``basedir``* directory, in a folder per patient. It is a simplified
version of the Dataset class in the `training
notebook <pytorch-monai-training.ipynb>`__.
notebook <pytorch-monai-training-with-output.html>`__.
Images are loaded with MONAI’s
`LoadImage <https://docs.monai.io/en/stable/transforms.html#loadimage>`__,
@ -313,8 +317,8 @@ method to display the images in the expected orientation:
def rotate_and_flip(image):
"""Rotate `image` by 90 degrees and flip horizontally"""
return cv2.flip(cv2.rotate(image, rotateCode=cv2.ROTATE_90_CLOCKWISE), flipCode=1)
class KitsDataset:
def __init__(self, basedir: str):
"""
@ -323,40 +327,40 @@ method to display the images in the expected orientation:
with each subdirectory containing directories imaging_frames, with jpg images, and
segmentation_frames with segmentation masks as png files.
See https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/110-ct-segmentation-quantize/data-preparation-ct-scan.ipynb
:param basedir: Directory that contains the prepared CT scans
"""
masks = sorted(BASEDIR.glob("case_*/segmentation_frames/*png"))
self.basedir = basedir
self.dataset = masks
print(
f"Created dataset with {len(self.dataset)} items. "
f"Base directory for data: {basedir}"
)
def __getitem__(self, index):
"""
Get an item from the dataset at the specified index.
:return: (image, segmentation_mask)
"""
mask_path = self.dataset[index]
image_path = str(mask_path.with_suffix(".jpg")).replace(
"segmentation_frames", "imaging_frames"
)
# Load images with MONAI's LoadImage to match data loading in training notebook
mask = LoadImage(image_only=True, dtype=np.uint8)(str(mask_path)).numpy()
img = LoadImage(image_only=True, dtype=np.float32)(str(image_path)).numpy()
if img.shape[:2] != (512, 512):
img = cv2.resize(img.astype(np.uint8), (512, 512)).astype(np.float32)
mask = cv2.resize(mask, (512, 512))
input_image = np.expand_dims(img, axis=0)
return input_image, mask
def __len__(self):
return len(self.dataset)
@ -374,10 +378,10 @@ kidney pixels to verify that the annotations look correct:
image_data, mask = next(item for item in dataset if np.count_nonzero(item[1]) > 5000)
# Remove extra image dimension and rotate and flip the image for visualization
image = rotate_and_flip(image_data.squeeze())
# The data loader returns annotations as (index, mask) and mask in shape (H,W)
mask = rotate_and_flip(mask)
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
ax[0].imshow(image, cmap="gray")
ax[1].imshow(mask, cmap="gray");
@ -455,7 +459,7 @@ this notebook.
.. code:: ipython3
fp32_ir_path = MODEL_DIR / Path('unet_kits19_fp32.xml')
fp32_ir_model = ov.convert_model(model, example_input=torch.ones(1, 1, 512, 512, dtype=torch.float32))
ov.save_model(fp32_ir_model, str(fp32_ir_path))
@ -468,8 +472,16 @@ this notebook.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
@ -501,8 +513,8 @@ steps:
"""
images, _ = data_item
return images
data_loader = torch.utils.data.DataLoader(dataset)
calibration_dataset = nncf.Dataset(data_loader, transform_fn)
quantized_model = nncf.quantize(
@ -527,10 +539,6 @@ steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -548,10 +556,7 @@ steps:
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -570,11 +575,11 @@ model and save it.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:333: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:334: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:341: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:342: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:179: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
@ -599,7 +604,7 @@ Compare File Size
fp32_ir_model_size = fp32_ir_path.with_suffix(".bin").stat().st_size / 1024
quantized_model_size = int8_ir_path.with_suffix(".bin").stat().st_size / 1024
print(f"FP32 IR model size: {fp32_ir_model_size:.2f} KB")
print(f"INT8 model size: {quantized_model_size:.2f} KB")
@ -607,7 +612,7 @@ Compare File Size
.. parsed-literal::
FP32 IR model size: 3864.14 KB
INT8 model size: 1940.55 KB
INT8 model size: 1940.41 KB
Compare Metrics for the original model and the quantized model to be sure that there no degradation.
@ -618,10 +623,10 @@ Compare Metrics for the original model and the quantized model to be sure that t
.. code:: ipython3
core = ov.Core()
int8_compiled_model = core.compile_model(int8_ir_model)
int8_f1 = compute_f1(int8_compiled_model, dataset)
print(f"FP32 F1: {fp32_f1:.3f}")
print(f"INT8 F1: {int8_f1:.3f}")
@ -672,32 +677,40 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
.. parsed-literal::
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 26.10 ms
[ INFO ] Read model took 26.68 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [?,?,?,?]
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add_425) : f32 / [...] / [?,1,16..,16..]
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [?,1,16..,16..]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [?,?,?,?]
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add_425) : f32 / [...] / [?,1,16..,16..]
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [?,1,16..,16..]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 80.42 ms
.. parsed-literal::
[ INFO ] Compile model took 86.15 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -705,9 +718,9 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] NUM_STREAMS: 1
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] PERF_COUNT: False
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
@ -719,9 +732,9 @@ be run in the notebook with ``! benchmark_app`` or
[Step 9/11] Creating infer requests and preparing input tensors
[ ERROR ] Input x is dynamic. Provide data shapes!
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 485, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 486, in main
data_queue = get_input_data(paths_to_input, app_inputs_info)
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
raise Exception(f"Input {info.name} is dynamic. Provide data shapes!")
Exception: Input x is dynamic. Provide data shapes!
@ -738,18 +751,18 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 12.76 ms
[ INFO ] Read model took 13.10 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x.1 (node: x.1) : f32 / [...] / [1,1,512,512]
@ -759,11 +772,19 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
.. parsed-literal::
[ INFO ] x.1 (node: x.1) : f32 / [N,C,H,W] / [1,1,512,512]
[ INFO ] Model outputs:
[ INFO ] 571 (node: 571) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 188.00 ms
.. parsed-literal::
[ INFO ] Compile model took 188.53 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: main_graph
@ -771,9 +792,9 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] NUM_STREAMS: 1
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] PERF_COUNT: False
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
@ -784,20 +805,28 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x.1'!. This input will be filled with random values!
[ INFO ] Fill input 'x.1' with random values
[ INFO ] Fill input 'x.1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 30.70 ms
.. parsed-literal::
[ INFO ] First inference took 30.02 ms
.. parsed-literal::
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 971 iterations
[ INFO ] Duration: 15006.86 ms
[ INFO ] Count: 964 iterations
[ INFO ] Duration: 15009.54 ms
[ INFO ] Latency:
[ INFO ] Median: 15.20 ms
[ INFO ] Average: 15.24 ms
[ INFO ] Min: 14.91 ms
[ INFO ] Max: 16.90 ms
[ INFO ] Throughput: 64.70 FPS
[ INFO ] Median: 15.33 ms
[ INFO ] Average: 15.36 ms
[ INFO ] Min: 14.97 ms
[ INFO ] Max: 17.10 ms
[ INFO ] Throughput: 64.23 FPS
Visually Compare Inference Results
@ -831,11 +860,11 @@ seed is displayed to enable reproducing specific runs of this cell.
# to binary segmentation masks
def sigmoid(x):
return np.exp(-np.logaddexp(0, -x))
num_images = 4
colormap = "gray"
# Load FP32 and INT8 models
core = ov.Core()
fp_model = core.read_model(fp32_ir_path)
@ -844,18 +873,18 @@ seed is displayed to enable reproducing specific runs of this cell.
compiled_model_int8 = core.compile_model(int8_model, device_name="CPU")
output_layer_fp = compiled_model_fp.output(0)
output_layer_int8 = compiled_model_int8.output(0)
# Create subset of dataset
background_slices = (item for item in dataset if np.count_nonzero(item[1]) == 0)
kidney_slices = (item for item in dataset if np.count_nonzero(item[1]) > 50)
data_subset = random.sample(list(background_slices), 2) + random.sample(list(kidney_slices), 2)
# Set seed to current time. To reproduce specific results, copy the printed seed
# and manually set `seed` to that value.
seed = int(time.time())
random.seed(seed)
print(f"Visualizing results with seed {seed}")
fig, ax = plt.subplots(nrows=num_images, ncols=4, figsize=(24, num_images * 4))
for i, (image, mask) in enumerate(data_subset):
display_image = rotate_and_flip(image.squeeze())
@ -864,13 +893,13 @@ seed is displayed to enable reproducing specific runs of this cell.
input_image = np.expand_dims(image, 0)
res_fp = compiled_model_fp([input_image])
res_int8 = compiled_model_int8([input_image])
# Process inference outputs and convert to binary segementation masks
result_mask_fp = sigmoid(res_fp[output_layer_fp]).squeeze().round().astype(np.uint8)
result_mask_int8 = sigmoid(res_int8[output_layer_int8]).squeeze().round().astype(np.uint8)
result_mask_fp = rotate_and_flip(result_mask_fp)
result_mask_int8 = rotate_and_flip(result_mask_int8)
# Display images, annotations, FP32 result and INT8 result
ax[i, 0].imshow(display_image, cmap=colormap)
ax[i, 1].imshow(target_mask, cmap=colormap)
@ -882,7 +911,7 @@ seed is displayed to enable reproducing specific runs of this cell.
.. parsed-literal::
Visualizing results with seed 1701899334
Visualizing results with seed 1706219510
@ -924,7 +953,7 @@ overlay of the segmentation mask on the original image/frame.
.. code:: ipython3
CASE = 117
segmentation_model = SegmentationModel(
ie=core, model_path=int8_ir_path, sigmoid=True, rotate_and_flip=True
)
@ -965,8 +994,8 @@ performs inference, and displays the results on the frames loaded in
.. parsed-literal::
Loaded model to CPU in 0.18 seconds.
Total time for 68 frames: 2.73 seconds, fps:25.30
Loaded model to CPU in 0.21 seconds.
Total time for 68 frames: 2.62 seconds, fps:26.34
References
@ -974,20 +1003,16 @@ References
**OpenVINO** - `NNCF
Repository <https://github.com/openvinotoolkit/nncf/>`__ - `Neural
Network Compression Framework for fast model
inference <https://arxiv.org/abs/2002.08679>`__ - `OpenVINO API
Tutorial <002-openvino-api-with-output.html>`__ - `OpenVINO
PyPI (pip install
openvino-dev) <https://pypi.org/project/openvino-dev/>`__
**OpenVINO**
**Kits19 Data** - `Kits19 Challenge
Homepage <https://kits19.grand-challenge.org/>`__ - `Kits19 GitHub
Repository <https://github.com/neheller/kits19>`__ - `The KiTS19
Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT
Semantic Segmentations, and Surgical
Outcomes <https://arxiv.org/abs/1904.00445>`__ - `The state of the art
in kidney and kidney tumor segmentation in contrast-enhanced CT imaging:
Results of the KiTS19
challenge <https://www.sciencedirect.com/science/article/pii/S1361841520301857>`__
- `NNCF Repository <https://github.com/openvinotoolkit/nncf/>`__
- `Neural Network Compression Framework for fast model inference <https://arxiv.org/abs/2002.08679>`__
- `OpenVINO API Tutorial <002-openvino-api-with-output.html>`__
- `OpenVINO PyPI (pip install openvino-dev) <https://pypi.org/project/openvino-dev/>`__
**Kits19 Data**
- `Kits19 Challenge Homepage <https://kits19.grand-challenge.org/>`__
- `Kits19 GitHub Repository <https://github.com/neheller/kits19>`__
- `The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes <https://arxiv.org/abs/1904.00445>`__
- `The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge <https://www.sciencedirect.com/science/article/pii/S1361841520301857>`__

4
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output_files/110-ct-segmentation-quantize-nncf-with-output_37_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:04761a5918e007435a0a6d8e1e44df408ba32326558935ebb4c47c48aa1d41cc
size 385394
oid sha256:8b31829da55a95766d9f5f4e8b93cec1e5bac3f1147fd2a5cea2e8d7f7b12abc
size 386985

9
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output_files/index.html
View File

@ -1,9 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-segmentation-quantize-nncf-with-output_15_1.png">110-ct-segmentation-quantize-nncf-with-output_1..&gt;</a> 07-Dec-2023 00:49 158997
<a href="110-ct-segmentation-quantize-nncf-with-output_37_1.png">110-ct-segmentation-quantize-nncf-with-output_3..&gt;</a> 07-Dec-2023 00:49 385394
<a href="110-ct-segmentation-quantize-nncf-with-output_42_0.jpg">110-ct-segmentation-quantize-nncf-with-output_4..&gt;</a> 07-Dec-2023 00:49 73812
</pre><hr></body>
</html>

9
docs/notebooks/110-ct-segmentation-quantize-nncf-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,9 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/110-ct-segmentation-quantize-nncf-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="110-ct-segmentation-quantize-nncf-with-output_15_1.png">110-ct-segmentation-quantize-nncf-with-output_1..&gt;</a> 26-Jan-2024 01:05 158997
<a href="110-ct-segmentation-quantize-nncf-with-output_37_1.png">110-ct-segmentation-quantize-nncf-with-output_3..&gt;</a> 26-Jan-2024 01:05 386985
<a href="110-ct-segmentation-quantize-nncf-with-output_42_0.jpg">110-ct-segmentation-quantize-nncf-with-output_4..&gt;</a> 26-Jan-2024 01:05 73812
</pre><hr></body>
</html>

1936
docs/notebooks/111-yolov5-quantization-migration-with-output.rst
View File

File diff suppressed because it is too large Load Diff

4
docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_34_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a7d7f50fd1250dd0675bd2dbaf57b27692cbc4648c9bd8caf0a90c9f303b7049
size 33667
oid sha256:cf96b8c432fd043883f1adf03e2e5f554812455814af6245b9a13eacf0e39696
size 33477

4
docs/notebooks/111-yolov5-quantization-migration-with-output_files/111-yolov5-quantization-migration-with-output_40_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:60c7a0fd81574ebcc9d02c1966c0c6dd30204e7c7de7cea02227faefc0b27708
size 770524
oid sha256:8bbf27a2ad6605dcb53802f06b329b5189988d317d7770f689d6ff8ba6b450e3
size 773347

8
docs/notebooks/111-yolov5-quantization-migration-with-output_files/index.html
View File

@ -1,8 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="111-yolov5-quantization-migration-with-output_34_0.png">111-yolov5-quantization-migration-with-output_3..&gt;</a> 31-Oct-2023 00:35 33667
<a href="111-yolov5-quantization-migration-with-output_40_0.png">111-yolov5-quantization-migration-with-output_4..&gt;</a> 31-Oct-2023 00:35 770524
</pre><hr></body>
</html>

8
docs/notebooks/111-yolov5-quantization-migration-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/111-yolov5-quantization-migration-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="111-yolov5-quantization-migration-with-output_34_0.png">111-yolov5-quantization-migration-with-output_3..&gt;</a> 26-Jan-2024 01:05 33477
<a href="111-yolov5-quantization-migration-with-output_40_0.png">111-yolov5-quantization-migration-with-output_4..&gt;</a> 26-Jan-2024 01:05 773347
</pre><hr></body>
</html>

418
docs/notebooks/112-pytorch-post-training-quantization-nncf-with-output.rst
View File

@ -22,8 +22,8 @@ quantization, not demanding the fine-tuning of the model.
the default binary search path of the OS you are running the
notebook.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Preparations <#preparations>`__
@ -41,14 +41,14 @@ quantization, not demanding the fine-tuning of the model.
- `Model quantization and
benchmarking <#model-quantization-and-benchmarking>`__
- `I. Evaluate the loaded model <#i-evaluate-the-loaded-model>`__
- `I. Evaluate the loaded model <#i--evaluate-the-loaded-model>`__
- `II. Create and initialize
quantization <#ii-create-and-initialize-quantization>`__
quantization <#ii--create-and-initialize-quantization>`__
- `III. Convert the models to OpenVINO Intermediate Representation
(OpenVINO
IR) <#iii-convert-the-models-to-openvino-intermediate-representation-openvino-ir>`__
IR) <#iii--convert-the-models-to-openvino-intermediate-representation-openvino-ir>`__
- `IV. Compare performance of INT8 model and FP32 model in
OpenVINO <#iv-compare-performance-of-int-model-and-fp-model-in-openvino>`__
OpenVINO <#iv--compare-performance-of-int8-model-and-fp32-model-in-openvino>`__
Preparations
------------
@ -65,6 +65,10 @@ Preparations
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -74,16 +78,16 @@ Preparations
# required C++ tools. This code assumes that Visual Studio 2019 is installed in the default
# directory. If you have a different C++ compiler, add the correct path to os.environ["PATH"]
# directly.
# Adding the path to os.environ["LIB"] is not always required - it depends on the system configuration.
import sys
if sys.platform == "win32":
import distutils.command.build_ext
import os
from pathlib import Path
VS_INSTALL_DIR = r"C:/Program Files (x86)/Microsoft Visual Studio"
cl_paths = sorted(list(Path(VS_INSTALL_DIR).glob("**/Hostx86/x64/cl.exe")))
if len(cl_paths) == 0:
@ -117,15 +121,15 @@ Imports
import zipfile
from pathlib import Path
from typing import List, Tuple
import nncf
import openvino as ov
import torch
from torchvision.datasets import ImageFolder
from torchvision.models import resnet50
import torchvision.transforms as transforms
sys.path.append("../utils")
from notebook_utils import download_file
@ -144,21 +148,21 @@ Settings
torch_device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using {torch_device} device")
MODEL_DIR = Path("model")
OUTPUT_DIR = Path("output")
BASE_MODEL_NAME = "resnet50"
IMAGE_SIZE = [64, 64]
OUTPUT_DIR.mkdir(exist_ok=True)
MODEL_DIR.mkdir(exist_ok=True)
# Paths where PyTorch and OpenVINO IR models will be stored.
fp32_checkpoint_filename = Path(BASE_MODEL_NAME + "_fp32").with_suffix(".pth")
fp32_ir_path = OUTPUT_DIR / Path(BASE_MODEL_NAME + "_fp32").with_suffix(".xml")
int8_ir_path = OUTPUT_DIR / Path(BASE_MODEL_NAME + "_int8").with_suffix(".xml")
fp32_pth_url = "https://storage.openvinotoolkit.org/repositories/nncf/openvino_notebook_ckpts/304_resnet50_fp32.pth"
download_file(fp32_pth_url, directory=MODEL_DIR, filename=fp32_checkpoint_filename)
@ -178,7 +182,7 @@ Settings
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/112-pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/112-pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
@ -204,29 +208,29 @@ Download and Prepare Tiny ImageNet dataset
zip_ref.extractall(path=output_dir)
zip_ref.close()
print(f"Successfully downloaded and extracted dataset to: {output_dir}")
def create_validation_dir(dataset_dir: Path):
VALID_DIR = dataset_dir / "val"
val_img_dir = VALID_DIR / "images"
fp = open(VALID_DIR / "val_annotations.txt", "r")
data = fp.readlines()
val_img_dict = {}
for line in data:
words = line.split("\t")
val_img_dict[words[0]] = words[1]
fp.close()
for img, folder in val_img_dict.items():
newpath = val_img_dir / folder
if not newpath.exists():
os.makedirs(newpath)
if (val_img_dir / img).exists():
os.rename(val_img_dir / img, newpath / img)
DATASET_DIR = OUTPUT_DIR / "tiny-imagenet-200"
if not DATASET_DIR.exists():
download_tiny_imagenet_200(OUTPUT_DIR)
@ -256,7 +260,7 @@ process.
class AverageMeter(object):
"""Computes and stores the average and current value"""
def __init__(self, name: str, fmt: str = ":f"):
self.name = name
self.fmt = fmt
@ -264,52 +268,52 @@ process.
self.avg = 0
self.sum = 0
self.count = 0
def update(self, val: float, n: int = 1):
self.val = val
self.sum += val * n
self.count += n
self.avg = self.sum / self.count
def __str__(self):
fmtstr = "{name} {val" + self.fmt + "} ({avg" + self.fmt + "})"
return fmtstr.format(**self.__dict__)
class ProgressMeter(object):
"""Displays the progress of validation process"""
def __init__(self, num_batches: int, meters: List[AverageMeter], prefix: str = ""):
self.batch_fmtstr = self._get_batch_fmtstr(num_batches)
self.meters = meters
self.prefix = prefix
def display(self, batch: int):
entries = [self.prefix + self.batch_fmtstr.format(batch)]
entries += [str(meter) for meter in self.meters]
print("\t".join(entries))
def _get_batch_fmtstr(self, num_batches: int):
num_digits = len(str(num_batches // 1))
fmt = "{:" + str(num_digits) + "d}"
return "[" + fmt + "/" + fmt.format(num_batches) + "]"
def accuracy(output: torch.Tensor, target: torch.Tensor, topk: Tuple[int] = (1,)):
"""Computes the accuracy over the k top predictions for the specified values of k"""
with torch.no_grad():
maxk = max(topk)
batch_size = target.size(0)
_, pred = output.topk(maxk, 1, True, True)
pred = pred.t()
correct = pred.eq(target.view(1, -1).expand_as(pred))
res = []
for k in topk:
correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)
res.append(correct_k.mul_(100.0 / batch_size))
return res
Validation function
@ -321,8 +325,8 @@ Validation function
from typing import Union
from openvino.runtime.ie_api import CompiledModel
def validate(val_loader: torch.utils.data.DataLoader, model: Union[torch.nn.Module, CompiledModel]):
"""Compute the metrics using data from val_loader for the model"""
batch_time = AverageMeter("Time", ":3.3f")
@ -334,13 +338,13 @@ Validation function
if not isinstance(model, CompiledModel):
model.eval()
model.to(torch_device)
with torch.no_grad():
end = time.time()
for i, (images, target) in enumerate(val_loader):
images = images.to(torch_device)
target = target.to(torch_device)
# Compute the output.
if isinstance(model, CompiledModel):
output_layer = model.output(0)
@ -348,20 +352,20 @@ Validation function
output = torch.from_numpy(output)
else:
output = model(images)
# Measure accuracy and record loss.
acc1, acc5 = accuracy(output, target, topk=(1, 5))
top1.update(acc1[0], images.size(0))
top5.update(acc5[0], images.size(0))
# Measure elapsed time.
batch_time.update(time.time() - end)
end = time.time()
print_frequency = 10
if i % print_frequency == 0:
progress.display(i)
print(
" * Acc@1 {top1.avg:.3f} Acc@5 {top5.avg:.3f} Total time: {total_time:.3f}".format(top1=top1, top5=top5, total_time=end - start_time)
)
@ -372,8 +376,7 @@ Create and load original uncompressed model
ResNet-50 from the `torchivision
repository <https://github.com/pytorch/vision>`__ is pre-trained on
ResNet-50 from the `torchivision repository <https://github.com/pytorch/vision>`__ is pre-trained on
ImageNet with more prediction classes than Tiny ImageNet, so the model
is adjusted by swapping the last FC layer to one with fewer output
values.
@ -393,8 +396,8 @@ values.
else:
raise RuntimeError("There is no checkpoint to load")
return model
model = create_model(MODEL_DIR / fp32_checkpoint_filename)
Create train and validation DataLoaders
@ -427,7 +430,7 @@ Create train and validation DataLoaders
[transforms.Resize(IMAGE_SIZE), transforms.ToTensor(), normalize]
),
)
train_loader = torch.utils.data.DataLoader(
train_dataset,
batch_size=batch_size,
@ -436,7 +439,7 @@ Create train and validation DataLoaders
pin_memory=True,
sampler=None,
)
val_loader = torch.utils.data.DataLoader(
val_dataset,
batch_size=batch_size,
@ -445,8 +448,8 @@ Create train and validation DataLoaders
pin_memory=True,
)
return train_loader, val_loader
train_loader, val_loader = create_dataloaders()
Model quantization and benchmarking
@ -471,15 +474,47 @@ I. Evaluate the loaded model
.. parsed-literal::
Test: [ 0/79] Time 0.257 (0.257) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.233 (0.231) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.223 (0.231) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.231 (0.231) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.232 (0.234) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.226 (0.233) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.225 (0.233) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.232 (0.234) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 18.296
Test: [ 0/79] Time 0.249 (0.249) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
.. parsed-literal::
Test: [10/79] Time 0.223 (0.231) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
.. parsed-literal::
Test: [20/79] Time 0.230 (0.231) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
.. parsed-literal::
Test: [30/79] Time 0.229 (0.230) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
.. parsed-literal::
Test: [40/79] Time 0.245 (0.229) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
.. parsed-literal::
Test: [50/79] Time 0.223 (0.229) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
.. parsed-literal::
Test: [60/79] Time 0.227 (0.229) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
.. parsed-literal::
Test: [70/79] Time 0.227 (0.230) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
.. parsed-literal::
* Acc@1 60.740 Acc@5 83.960 Total time: 18.038
Test accuracy of FP32 model: 60.740
@ -495,7 +530,7 @@ layers. The framework is designed so that modifications to your original
training code are minor. Quantization is the simplest scenario and
requires a few modifications. For more information about NNCF Post
Training Quantization (PTQ) API, refer to the `Basic Quantization Flow
Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-qauntization-flow>`__.
Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html>`__.
1. Create a transformation function that accepts a sample from the
dataset and returns data suitable for model inference. This enables
@ -508,8 +543,8 @@ Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-
def transform_fn(data_item):
images, _ = data_item
return images
calibration_dataset = nncf.Dataset(train_loader, transform_fn)
2. Create a quantized model from the pre-trained ``FP32`` model and the
@ -522,10 +557,23 @@ Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-
.. parsed-literal::
2023-12-06 22:54:25.232060: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-06 22:54:25.263296: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 22:57:45.269741: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 22:57:45.300230: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-06 22:54:25.771528: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 22:57:45.813067: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.1.0+cpu. If you encounter issues, consider switching to torch==2.1.2
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
@ -543,16 +591,16 @@ Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
INFO:nncf:Compiling and loading torch extension: quantized_functions_cpu...
.. parsed-literal::
INFO:nncf:Finished loading torch extension: quantized_functions_cpu
@ -570,10 +618,6 @@ Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
@ -590,16 +634,48 @@ Guide <https://docs.openvino.ai/2023.3/basic_quantization_flow.html#doxid-basic-
.. parsed-literal::
Test: [ 0/79] Time 0.407 (0.407) Acc@1 82.03 (82.03) Acc@5 91.41 (91.41)
Test: [10/79] Time 0.387 (0.389) Acc@1 54.69 (66.69) Acc@5 85.16 (87.43)
Test: [20/79] Time 0.387 (0.387) Acc@1 67.97 (63.99) Acc@5 84.38 (87.17)
Test: [30/79] Time 0.386 (0.386) Acc@1 53.12 (62.42) Acc@5 77.34 (84.93)
Test: [40/79] Time 0.385 (0.385) Acc@1 66.41 (60.96) Acc@5 90.62 (84.24)
Test: [50/79] Time 0.385 (0.386) Acc@1 58.59 (60.71) Acc@5 88.28 (84.18)
Test: [60/79] Time 0.387 (0.386) Acc@1 65.62 (60.26) Acc@5 85.94 (83.62)
Test: [70/79] Time 0.385 (0.386) Acc@1 53.12 (60.00) Acc@5 80.47 (83.16)
* Acc@1 60.450 Acc@5 83.800 Total time: 30.199
Accuracy of initialized INT8 model: 60.450
Test: [ 0/79] Time 0.430 (0.430) Acc@1 81.25 (81.25) Acc@5 89.84 (89.84)
.. parsed-literal::
Test: [10/79] Time 0.392 (0.397) Acc@1 56.25 (66.26) Acc@5 85.16 (87.14)
.. parsed-literal::
Test: [20/79] Time 0.393 (0.395) Acc@1 68.75 (63.80) Acc@5 84.38 (86.98)
.. parsed-literal::
Test: [30/79] Time 0.393 (0.396) Acc@1 52.34 (62.15) Acc@5 75.78 (85.01)
.. parsed-literal::
Test: [40/79] Time 0.392 (0.395) Acc@1 67.19 (60.75) Acc@5 89.84 (84.26)
.. parsed-literal::
Test: [50/79] Time 0.392 (0.395) Acc@1 57.81 (60.68) Acc@5 88.28 (84.15)
.. parsed-literal::
Test: [60/79] Time 0.393 (0.394) Acc@1 64.06 (60.32) Acc@5 85.94 (83.58)
.. parsed-literal::
Test: [70/79] Time 0.392 (0.394) Acc@1 54.69 (60.17) Acc@5 78.12 (83.08)
.. parsed-literal::
* Acc@1 60.690 Acc@5 83.740 Total time: 30.882
Accuracy of initialized INT8 model: 60.690
It should be noted that the inference time for the quantized PyTorch
@ -623,9 +699,9 @@ For more information about model conversion, refer to this
.. code:: ipython3
dummy_input = torch.randn(128, 3, *IMAGE_SIZE)
model_ir = ov.convert_model(model, example_input=dummy_input, input=[-1, 3, *IMAGE_SIZE])
ov.save_model(model_ir, fp32_ir_path)
@ -642,22 +718,26 @@ For more information about model conversion, refer to this
.. code:: ipython3
quantized_model_ir = ov.convert_model(quantized_model, example_input=dummy_input, input=[-1, 3, *IMAGE_SIZE])
ov.save_model(quantized_model_ir, int8_ir_path)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:333: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:334: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:341: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:342: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!
Mismatched elements: 25580 / 25600 (99.9%)
Greatest absolute difference: 0.45703113079071045 at index (99, 85) (up to 1e-05 allowed)
Greatest relative difference: 134.48766541727224 at index (92, 158) (up to 1e-05 allowed)
Mismatched elements: 25573 / 25600 (99.9%)
Greatest absolute difference: 0.5424436330795288 at index (1, 149) (up to 1e-05 allowed)
Greatest relative difference: 42.99047422811133 at index (90, 158) (up to 1e-05 allowed)
_check_trace(
@ -666,7 +746,7 @@ Select inference device for OpenVINO
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
@ -674,7 +754,7 @@ Select inference device for OpenVINO
description='Device:',
disabled=False,
)
device
@ -698,15 +778,47 @@ Evaluate the FP32 and INT8 models.
.. parsed-literal::
Test: [ 0/79] Time 0.192 (0.192) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.135 (0.142) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
Test: [20/79] Time 0.138 (0.140) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
Test: [30/79] Time 0.136 (0.139) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
Test: [40/79] Time 0.138 (0.139) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
Test: [50/79] Time 0.138 (0.138) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
Test: [60/79] Time 0.137 (0.138) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
Test: [70/79] Time 0.138 (0.138) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
* Acc@1 60.740 Acc@5 83.960 Total time: 10.797
Test: [ 0/79] Time 0.184 (0.184) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
.. parsed-literal::
Test: [10/79] Time 0.139 (0.143) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
.. parsed-literal::
Test: [20/79] Time 0.139 (0.141) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
.. parsed-literal::
Test: [30/79] Time 0.139 (0.140) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
.. parsed-literal::
Test: [40/79] Time 0.135 (0.140) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
.. parsed-literal::
Test: [50/79] Time 0.139 (0.140) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
.. parsed-literal::
Test: [60/79] Time 0.139 (0.139) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
.. parsed-literal::
Test: [70/79] Time 0.138 (0.139) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
.. parsed-literal::
* Acc@1 60.740 Acc@5 83.960 Total time: 10.882
Accuracy of FP32 IR model: 60.740
@ -719,16 +831,48 @@ Evaluate the FP32 and INT8 models.
.. parsed-literal::
Test: [ 0/79] Time 0.138 (0.138) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
Test: [10/79] Time 0.077 (0.082) Acc@1 53.91 (66.83) Acc@5 85.94 (87.36)
Test: [20/79] Time 0.076 (0.079) Acc@1 67.19 (64.10) Acc@5 83.59 (87.02)
Test: [30/79] Time 0.076 (0.078) Acc@1 53.12 (62.15) Acc@5 76.56 (84.95)
Test: [40/79] Time 0.076 (0.078) Acc@1 67.19 (60.71) Acc@5 89.06 (84.22)
Test: [50/79] Time 0.074 (0.077) Acc@1 58.59 (60.68) Acc@5 88.28 (84.19)
Test: [60/79] Time 0.080 (0.077) Acc@1 65.62 (60.31) Acc@5 87.50 (83.70)
Test: [70/79] Time 0.074 (0.077) Acc@1 53.12 (60.08) Acc@5 79.69 (83.23)
* Acc@1 60.620 Acc@5 83.890 Total time: 6.027
Accuracy of INT8 IR model: 60.620
Test: [ 0/79] Time 0.133 (0.133) Acc@1 81.25 (81.25) Acc@5 91.41 (91.41)
.. parsed-literal::
Test: [10/79] Time 0.076 (0.084) Acc@1 54.69 (66.41) Acc@5 85.94 (87.64)
.. parsed-literal::
Test: [20/79] Time 0.077 (0.080) Acc@1 71.09 (64.10) Acc@5 84.38 (87.05)
.. parsed-literal::
Test: [30/79] Time 0.075 (0.079) Acc@1 52.34 (62.17) Acc@5 75.00 (84.98)
.. parsed-literal::
Test: [40/79] Time 0.077 (0.079) Acc@1 67.19 (60.67) Acc@5 89.84 (84.22)
.. parsed-literal::
Test: [50/79] Time 0.077 (0.078) Acc@1 60.16 (60.63) Acc@5 88.28 (84.22)
.. parsed-literal::
Test: [60/79] Time 0.076 (0.078) Acc@1 65.62 (60.31) Acc@5 86.72 (83.67)
.. parsed-literal::
Test: [70/79] Time 0.076 (0.077) Acc@1 53.12 (60.01) Acc@5 78.91 (83.20)
.. parsed-literal::
* Acc@1 60.540 Acc@5 83.840 Total time: 6.060
Accuracy of INT8 IR model: 60.540
IV. Compare performance of INT8 model and FP32 model in OpenVINO
@ -771,20 +915,20 @@ throughput (frames per second) values.
"""Prints the output from benchmark_app in human-readable format"""
parsed_output = [line for line in benchmark_output if 'FPS' in line]
print(*parsed_output, sep='\n')
print('Benchmark FP32 model (OpenVINO IR)')
benchmark_output = ! benchmark_app -m "$fp32_ir_path" -d $device.value -api async -t 15 -shape "[1, 3, 512, 512]"
parse_benchmark_output(benchmark_output)
print('Benchmark INT8 model (OpenVINO IR)')
benchmark_output = ! benchmark_app -m "$int8_ir_path" -d $device.value -api async -t 15 -shape "[1, 3, 512, 512]"
parse_benchmark_output(benchmark_output)
print('Benchmark FP32 model (OpenVINO IR) synchronously')
benchmark_output = ! benchmark_app -m "$fp32_ir_path" -d $device.value -api sync -t 15 -shape "[1, 3, 512, 512]"
parse_benchmark_output(benchmark_output)
print('Benchmark INT8 model (OpenVINO IR) synchronously')
benchmark_output = ! benchmark_app -m "$int8_ir_path" -d $device.value -api sync -t 15 -shape "[1, 3, 512, 512]"
parse_benchmark_output(benchmark_output)
@ -793,13 +937,29 @@ throughput (frames per second) values.
.. parsed-literal::
Benchmark FP32 model (OpenVINO IR)
[ INFO ] Throughput: 39.12 FPS
.. parsed-literal::
[ INFO ] Throughput: 38.82 FPS
Benchmark INT8 model (OpenVINO IR)
[ INFO ] Throughput: 155.73 FPS
.. parsed-literal::
[ INFO ] Throughput: 157.27 FPS
Benchmark FP32 model (OpenVINO IR) synchronously
[ INFO ] Throughput: 40.37 FPS
.. parsed-literal::
[ INFO ] Throughput: 40.19 FPS
Benchmark INT8 model (OpenVINO IR) synchronously
[ INFO ] Throughput: 137.36 FPS
.. parsed-literal::
[ INFO ] Throughput: 137.35 FPS
Show device Information for reference:
@ -808,7 +968,7 @@ Show device Information for reference:
core = ov.Core()
devices = core.available_devices
for device_name in devices:
device_full_name = core.get_property(device_name, "FULL_DEVICE_NAME")
print(f"{device_name}: {device_full_name}")

2770
docs/notebooks/113-image-classification-quantization-with-output.rst
View File

File diff suppressed because it is too large Load Diff

0
docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_2.png → docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_5.png
View File

7
docs/notebooks/113-image-classification-quantization-with-output_files/index.html
View File

@ -1,7 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/113-image-classification-quantization-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/113-image-classification-quantization-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="113-image-classification-quantization-with-output_30_2.png">113-image-classification-quantization-with-outp..&gt;</a> 07-Dec-2023 00:49 14855
</pre><hr></body>
</html>

7
docs/notebooks/113-image-classification-quantization-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/113-image-classification-quantization-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/113-image-classification-quantization-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="113-image-classification-quantization-with-output_30_5.png">113-image-classification-quantization-with-outp..&gt;</a> 26-Jan-2024 01:05 14855
</pre><hr></body>
</html>

215
docs/notebooks/115-async-api-with-output.rst
View File

@ -11,8 +11,8 @@ device is busy with inference, the application can perform other tasks
in parallel (for example, populating inputs or scheduling other
requests) rather than wait for the current inference to complete first.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Prepare model and data
@ -37,7 +37,8 @@ requests) rather than wait for the current inference to complete first.
- `AsyncInferQueue <#asyncinferqueue>`__
- `Setting Callback <#setting-callback>`__
- `Test the performance with AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__
- `Test the performance with
AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__
Imports
-------
@ -53,6 +54,10 @@ Imports
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -113,10 +118,204 @@ each frame of the video.
################|| Downloading person-detection-0202 ||################
========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.xml
.. parsed-literal::
... 12%, 32 KB, 1452 KB/s, 0 seconds passed
.. parsed-literal::
... 25%, 64 KB, 1385 KB/s, 0 seconds passed
... 38%, 96 KB, 1632 KB/s, 0 seconds passed
.. parsed-literal::
... 51%, 128 KB, 1424 KB/s, 0 seconds passed
... 64%, 160 KB, 1393 KB/s, 0 seconds passed
... 77%, 192 KB, 1664 KB/s, 0 seconds passed
... 89%, 224 KB, 1795 KB/s, 0 seconds passed
... 100%, 248 KB, 1986 KB/s, 0 seconds passed
.. parsed-literal::
========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.bin
.. parsed-literal::
... 0%, 32 KB, 1284 KB/s, 0 seconds passed
... 1%, 64 KB, 1284 KB/s, 0 seconds passed
... 2%, 96 KB, 1894 KB/s, 0 seconds passed
.. parsed-literal::
... 3%, 128 KB, 1699 KB/s, 0 seconds passed
... 4%, 160 KB, 2092 KB/s, 0 seconds passed
... 5%, 192 KB, 2274 KB/s, 0 seconds passed
... 6%, 224 KB, 2623 KB/s, 0 seconds passed
... 7%, 256 KB, 2328 KB/s, 0 seconds passed
... 8%, 288 KB, 2599 KB/s, 0 seconds passed
.. parsed-literal::
... 9%, 320 KB, 2693 KB/s, 0 seconds passed
... 9%, 352 KB, 2939 KB/s, 0 seconds passed
... 10%, 384 KB, 2655 KB/s, 0 seconds passed
... 11%, 416 KB, 2858 KB/s, 0 seconds passed
... 12%, 448 KB, 2918 KB/s, 0 seconds passed
... 13%, 480 KB, 3100 KB/s, 0 seconds passed
.. parsed-literal::
... 14%, 512 KB, 2858 KB/s, 0 seconds passed
... 15%, 544 KB, 2904 KB/s, 0 seconds passed
... 16%, 576 KB, 3065 KB/s, 0 seconds passed
... 17%, 608 KB, 3219 KB/s, 0 seconds passed
... 18%, 640 KB, 2998 KB/s, 0 seconds passed
... 18%, 672 KB, 3133 KB/s, 0 seconds passed
.. parsed-literal::
... 19%, 704 KB, 3164 KB/s, 0 seconds passed
... 20%, 736 KB, 3293 KB/s, 0 seconds passed
... 21%, 768 KB, 3098 KB/s, 0 seconds passed
... 22%, 800 KB, 3120 KB/s, 0 seconds passed
... 23%, 832 KB, 3228 KB/s, 0 seconds passed
.. parsed-literal::
... 24%, 864 KB, 3062 KB/s, 0 seconds passed
... 25%, 896 KB, 3170 KB/s, 0 seconds passed
... 26%, 928 KB, 3274 KB/s, 0 seconds passed
... 27%, 960 KB, 3291 KB/s, 0 seconds passed
... 27%, 992 KB, 3388 KB/s, 0 seconds passed
... 28%, 1024 KB, 3231 KB/s, 0 seconds passed
... 29%, 1056 KB, 3322 KB/s, 0 seconds passed
.. parsed-literal::
... 30%, 1088 KB, 3331 KB/s, 0 seconds passed
... 31%, 1120 KB, 3189 KB/s, 0 seconds passed
... 32%, 1152 KB, 3275 KB/s, 0 seconds passed
... 33%, 1184 KB, 3360 KB/s, 0 seconds passed
... 34%, 1216 KB, 3370 KB/s, 0 seconds passed
... 35%, 1248 KB, 3448 KB/s, 0 seconds passed
.. parsed-literal::
... 36%, 1280 KB, 3316 KB/s, 0 seconds passed
... 36%, 1312 KB, 3390 KB/s, 0 seconds passed
... 37%, 1344 KB, 3401 KB/s, 0 seconds passed
... 38%, 1376 KB, 3472 KB/s, 0 seconds passed
... 39%, 1408 KB, 3349 KB/s, 0 seconds passed
... 40%, 1440 KB, 3418 KB/s, 0 seconds passed
.. parsed-literal::
... 41%, 1472 KB, 3426 KB/s, 0 seconds passed
... 42%, 1504 KB, 3488 KB/s, 0 seconds passed
... 43%, 1536 KB, 3372 KB/s, 0 seconds passed
... 44%, 1568 KB, 3437 KB/s, 0 seconds passed
... 45%, 1600 KB, 3446 KB/s, 0 seconds passed
... 45%, 1632 KB, 3505 KB/s, 0 seconds passed
.. parsed-literal::
... 46%, 1664 KB, 3400 KB/s, 0 seconds passed
... 47%, 1696 KB, 3457 KB/s, 0 seconds passed
... 48%, 1728 KB, 3468 KB/s, 0 seconds passed
... 49%, 1760 KB, 3520 KB/s, 0 seconds passed
... 50%, 1792 KB, 3417 KB/s, 0 seconds passed
... 51%, 1824 KB, 3472 KB/s, 0 seconds passed
.. parsed-literal::
... 52%, 1856 KB, 3478 KB/s, 0 seconds passed
... 53%, 1888 KB, 3383 KB/s, 0 seconds passed
... 54%, 1920 KB, 3437 KB/s, 0 seconds passed
... 54%, 1952 KB, 3487 KB/s, 0 seconds passed
... 55%, 1984 KB, 3495 KB/s, 0 seconds passed
... 56%, 2016 KB, 3544 KB/s, 0 seconds passed
.. parsed-literal::
... 57%, 2048 KB, 3451 KB/s, 0 seconds passed
... 58%, 2080 KB, 3500 KB/s, 0 seconds passed
... 59%, 2112 KB, 3506 KB/s, 0 seconds passed
... 60%, 2144 KB, 3554 KB/s, 0 seconds passed
... 61%, 2176 KB, 3466 KB/s, 0 seconds passed
.. parsed-literal::
... 62%, 2208 KB, 3510 KB/s, 0 seconds passed
... 63%, 2240 KB, 3513 KB/s, 0 seconds passed
... 64%, 2272 KB, 3561 KB/s, 0 seconds passed
... 64%, 2304 KB, 3478 KB/s, 0 seconds passed
... 65%, 2336 KB, 3518 KB/s, 0 seconds passed
... 66%, 2368 KB, 3521 KB/s, 0 seconds passed
.. parsed-literal::
... 67%, 2400 KB, 3444 KB/s, 0 seconds passed
... 68%, 2432 KB, 3486 KB/s, 0 seconds passed
... 69%, 2464 KB, 3495 KB/s, 0 seconds passed
... 70%, 2496 KB, 3528 KB/s, 0 seconds passed
.. parsed-literal::
... 71%, 2528 KB, 3454 KB/s, 0 seconds passed
... 72%, 2560 KB, 3496 KB/s, 0 seconds passed
... 73%, 2592 KB, 3505 KB/s, 0 seconds passed
... 73%, 2624 KB, 3515 KB/s, 0 seconds passed
... 74%, 2656 KB, 3463 KB/s, 0 seconds passed
... 75%, 2688 KB, 3502 KB/s, 0 seconds passed
... 76%, 2720 KB, 3514 KB/s, 0 seconds passed
... 77%, 2752 KB, 3523 KB/s, 0 seconds passed
.. parsed-literal::
... 78%, 2784 KB, 3474 KB/s, 0 seconds passed
... 79%, 2816 KB, 3510 KB/s, 0 seconds passed
... 80%, 2848 KB, 3522 KB/s, 0 seconds passed
... 81%, 2880 KB, 3530 KB/s, 0 seconds passed
.. parsed-literal::
... 82%, 2912 KB, 3482 KB/s, 0 seconds passed
... 82%, 2944 KB, 3518 KB/s, 0 seconds passed
... 83%, 2976 KB, 3529 KB/s, 0 seconds passed
... 84%, 3008 KB, 3535 KB/s, 0 seconds passed
... 85%, 3040 KB, 3489 KB/s, 0 seconds passed
... 86%, 3072 KB, 3523 KB/s, 0 seconds passed
... 87%, 3104 KB, 3532 KB/s, 0 seconds passed
.. parsed-literal::
... 88%, 3136 KB, 3540 KB/s, 0 seconds passed
... 89%, 3168 KB, 3498 KB/s, 0 seconds passed
... 90%, 3200 KB, 3530 KB/s, 0 seconds passed
... 91%, 3232 KB, 3539 KB/s, 0 seconds passed
... 91%, 3264 KB, 3544 KB/s, 0 seconds passed
.. parsed-literal::
... 92%, 3296 KB, 3504 KB/s, 0 seconds passed
... 93%, 3328 KB, 3536 KB/s, 0 seconds passed
... 94%, 3360 KB, 3544 KB/s, 0 seconds passed
... 95%, 3392 KB, 3550 KB/s, 0 seconds passed
... 96%, 3424 KB, 3509 KB/s, 0 seconds passed
... 97%, 3456 KB, 3540 KB/s, 0 seconds passed
... 98%, 3488 KB, 3548 KB/s, 0 seconds passed
.. parsed-literal::
... 99%, 3520 KB, 3560 KB/s, 0 seconds passed
... 100%, 3549 KB, 3521 KB/s, 1 seconds passed
@ -313,7 +512,7 @@ Test performance in Sync Mode
.. parsed-literal::
Source ended
average throuput in sync mode: 40.67 fps
average throuput in sync mode: 43.51 fps
Async Mode
@ -452,7 +651,7 @@ Test the performance in Async Mode
.. parsed-literal::
Source ended
average throuput in async mode: 74.75 fps
average throuput in async mode: 74.01 fps
Compare the performance
@ -595,5 +794,5 @@ Test the performance with ``AsyncInferQueue``
.. parsed-literal::
average throughput in async mode with async infer queue: 111.75 fps
average throughput in async mode with async infer queue: 110.03 fps

4
docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8b8c8b61f0bbb25c280a3e72cf2172fd29bf11668231cb3d2527c1b8a05307f2
size 30406
oid sha256:a6ef5964af038dfaf0c794dfc9a36258293ea7a6f8318f5f84490e8953a10479
size 30402

10
docs/notebooks/115-async-api-with-output_files/index.html
View File

@ -1,10 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/115-async-api-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/115-async-api-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="115-async-api-with-output_15_0.png">115-async-api-with-output_15_0.png</a> 07-Dec-2023 00:49 4307
<a href="115-async-api-with-output_19_0.png">115-async-api-with-output_19_0.png</a> 07-Dec-2023 00:49 4307
<a href="115-async-api-with-output_21_0.png">115-async-api-with-output_21_0.png</a> 07-Dec-2023 00:49 30406
<a href="115-async-api-with-output_27_0.png">115-async-api-with-output_27_0.png</a> 07-Dec-2023 00:49 4307
</pre><hr></body>
</html>

10
docs/notebooks/115-async-api-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,10 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/115-async-api-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/115-async-api-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="115-async-api-with-output_15_0.png">115-async-api-with-output_15_0.png</a> 26-Jan-2024 01:05 4307
<a href="115-async-api-with-output_19_0.png">115-async-api-with-output_19_0.png</a> 26-Jan-2024 01:05 4307
<a href="115-async-api-with-output_21_0.png">115-async-api-with-output_21_0.png</a> 26-Jan-2024 01:05 30402
<a href="115-async-api-with-output_27_0.png">115-async-api-with-output_27_0.png</a> 26-Jan-2024 01:05 4307
</pre><hr></body>
</html>

157
docs/notebooks/116-sparsity-optimization-with-output.rst
View File

@ -12,7 +12,7 @@ datasets <https://huggingface.co/datasets/sst2>`__ using
`Optimum-Intel <https://github.com/huggingface/optimum-intel>`__. It
demonstrates the inference performance advantage on 4th Gen Intel® Xeon®
Scalable Processors by running it with `Sparse Weight
Decompression <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_CPU.html#sparse-weights-decompression>`__,
Decompression <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_CPU.html#sparse-weights-decompression-intel-x86-64>`__,
a runtime option that seizes model sparsity for efficiency. The notebook
consists of the following steps:
@ -21,8 +21,8 @@ consists of the following steps:
integration with Hugging Face Optimum.
- Compare sparse 8-bit vs. dense 8-bit inference performance.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
@ -51,6 +51,10 @@ Prerequisites
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -77,10 +81,18 @@ Imports
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-12-06 23:02:39.282111: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-06 23:02:39.316382: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
.. parsed-literal::
2024-01-25 23:06:01.802535: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 23:06:01.837209: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-06 23:02:40.030243: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 23:06:02.398511: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Download, quantize and sparsify the model, using Hugging Face Optimum API
@ -117,7 +129,11 @@ model card on Hugging Face.
.. parsed-literal::
Compiling the model to CPU ...
Setting OpenVINO CACHE_DIR to /opt/home/k8sworker/.cache/huggingface/hub/models--OpenVINO--bert-base-uncased-sst2-int8-unstructured80/snapshots/dc44eb46300882463d50ee847e0f6485bad3cdad/model_cache
.. parsed-literal::
device must be of type <class 'str'> but got <class 'torch.device'> instead
.. parsed-literal::
@ -193,18 +209,26 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
.. parsed-literal::
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 60.26 ms
.. parsed-literal::
[ INFO ] Read model took 60.22 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@ -215,7 +239,7 @@ as an example. It is recommended to tune based on your applications.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
[ INFO ] Reshape model took 24.75 ms
[ INFO ] Reshape model took 23.06 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@ -224,7 +248,11 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Model outputs:
[ INFO ] logits (node: logits) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1092.05 ms
.. parsed-literal::
[ INFO ] Compile model took 1064.23 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@ -232,9 +260,9 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] NUM_STREAMS: 4
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 4
[ INFO ] PERF_COUNT: False
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
@ -252,17 +280,25 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 27.56 ms
.. parsed-literal::
[ INFO ] First inference took 27.61 ms
.. parsed-literal::
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 8952 iterations
[ INFO ] Duration: 60029.00 ms
[ INFO ] Count: 8900 iterations
[ INFO ] Duration: 60039.72 ms
[ INFO ] Latency:
[ INFO ] Median: 26.46 ms
[ INFO ] Average: 26.52 ms
[ INFO ] Min: 25.37 ms
[ INFO ] Max: 40.49 ms
[ INFO ] Throughput: 149.13 FPS
[ INFO ] Median: 26.68 ms
[ INFO ] Average: 26.74 ms
[ INFO ] Min: 25.09 ms
[ INFO ] Max: 39.62 ms
[ INFO ] Throughput: 148.24 FPS
Benchmark quantized sparse inference performance
@ -308,18 +344,26 @@ for which a layer will be enabled.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
.. parsed-literal::
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 61.56 ms
.. parsed-literal::
[ INFO ] Read model took 67.79 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@ -330,7 +374,11 @@ for which a layer will be enabled.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
[ INFO ] Reshape model took 24.68 ms
.. parsed-literal::
[ INFO ] Reshape model took 23.92 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@ -339,45 +387,20 @@ for which a layer will be enabled.
[ INFO ] Model outputs:
[ INFO ] logits (node: logits) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1029.24 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ] NUM_STREAMS: 4
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 4
[ INFO ] PERF_COUNT: False
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 0.75
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'attention_mask'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
[ INFO ] Fill input 'input_ids' with random values
[ INFO ] Fill input 'attention_mask' with random values
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 29.95 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 8984 iterations
[ INFO ] Duration: 60026.81 ms
[ INFO ] Latency:
[ INFO ] Median: 26.52 ms
[ INFO ] Average: 26.59 ms
[ INFO ] Min: 23.89 ms
[ INFO ] Max: 40.95 ms
[ INFO ] Throughput: 149.67 FPS
[ ERROR ] Exception from src/inference/src/core.cpp:99:
[ GENERAL_ERROR ] Exception from src/plugins/intel_cpu/src/config.cpp:158:
Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 408, in main
compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config)
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/runtime/ie_api.py", line 547, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/core.cpp:99:
[ GENERAL_ERROR ] Exception from src/plugins/intel_cpu/src/config.cpp:158:
Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers
When this might be helpful
@ -394,6 +417,6 @@ For more details about asynchronous inference with OpenVINO, refer to
the following documentation:
- `Deployment Optimization
Guide <https://docs.openvino.ai/2023.3/openvino_docs_deployment_optimization_guide_common.html#doxid-openvino-docs-deployment-optimization-guide-common-1async-api>`__
Guide <https://docs.openvino.ai/2023.3/openvino_docs_deployment_optimization_guide_common.html>`__
- `Inference Request
API <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Infer_request.html#doxid-openvino-docs-o-v-u-g-infer-request-1in-out-tensors>`__
API <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Infer_request.html>`__

272
docs/notebooks/117-model-server-with-output.rst
View File

@ -34,18 +34,18 @@ deployment:
ovms_diagram
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Serving with OpenVINO Model
Server <#serving-with-openvino-model-server>`__
- `Step 1: Prepare Docker <#step--prepare-docker>`__
- `Step 1: Prepare Docker <#step-1-prepare-docker>`__
- `Step 2: Preparing a Model
Repository <#step--preparing-a-model-repository>`__
Repository <#step-2-preparing-a-model-repository>`__
- `Step 3: Start the Model Server
Container <#step--start-the-model-server-container>`__
Container <#step-3-start-the-model-server-container>`__
- `Step 4: Prepare the Example Client
Components <#step--prepare-the-example-client-components>`__
Components <#step-4-prepare-the-example-client-components>`__
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
@ -58,15 +58,16 @@ deployment:
- `References <#references>`__
Serving with OpenVINO Model Server
----------------------------------------------------------------------------
Serving with OpenVINO Model Server
----------------------------------
OpenVINO Model Server (OVMS) is a high-performance system for serving
models. Implemented in C++ for scalability and optimized for deployment
on Intel architectures, the model server uses the same architecture and
API as TensorFlow Serving and KServe while applying OpenVINO for
inference execution. Inference service is provided via gRPC or REST API,
making deploying new algorithms and AI experiments easy.
OpenVINO Model Server (OVMS) is
a high-performance system for serving models. Implemented in C++ for
scalability and optimized for deployment on Intel architectures, the
model server uses the same architecture and API as TensorFlow Serving
and KServe while applying OpenVINO for inference execution. Inference
service is provided via gRPC or REST API, making deploying new
algorithms and AI experiments easy.
.. figure:: https://user-images.githubusercontent.com/91237924/215658767-0e0fc221-aed0-4db1-9a82-6be55f244dba.png
:alt: ovms_high_level
@ -75,11 +76,11 @@ making deploying new algorithms and AI experiments easy.
To quickly start using OpenVINO™ Model Server, follow these steps:
Step 1: Prepare Docker
----------------------------------------------------------------
Step 1: Prepare Docker
----------------------
Install `Docker Engine <https://docs.docker.com/engine/install/>`__,
including its
Install `Docker
Engine <https://docs.docker.com/engine/install/>`__, including its
`post-installation <https://docs.docker.com/engine/install/linux-postinstall/>`__
steps, on your development system. To verify installation, test it,
using the following command. When it is ready, it will display a test
@ -92,10 +93,10 @@ image and a message.
.. parsed-literal::
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
@ -104,23 +105,24 @@ image and a message.
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
Step 2: Preparing a Model Repository
------------------------------------------------------------------------------
The models need to be placed and mounted in a particular directory
structure and according to the following rules:
Step 2: Preparing a Model Repository
------------------------------------
The models need to be placed
and mounted in a particular directory structure and according to the
following rules:
::
@ -173,7 +175,7 @@ structure and according to the following rules:
.. code:: ipython3
import os
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -181,40 +183,76 @@ structure and according to the following rules:
filename='notebook_utils.py'
)
from notebook_utils import download_file
dedicated_dir = "models"
model_name = "detection"
model_version = "1"
MODEL_DIR = f"{dedicated_dir}/{model_name}/{model_version}"
XML_PATH = "horizontal-text-detection-0001.xml"
BIN_PATH = "horizontal-text-detection-0001.bin"
os.makedirs(MODEL_DIR, exist_ok=True)
model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.xml"
model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.bin"
download_file(model_xml_url, XML_PATH, MODEL_DIR)
download_file(model_bin_url, BIN_PATH, MODEL_DIR)
.. parsed-literal::
Model Copied to "./models/detection/1".
models/detection/1/horizontal-text-detection-0001.xml: 0%| | 0.00/680k [00:00<?, ?B/s]
Step 3: Start the Model Server Container
----------------------------------------------------------------------------------
.. parsed-literal::
models/detection/1/horizontal-text-detection-0001.bin: 0%| | 0.00/7.39M [00:00<?, ?B/s]
.. parsed-literal::
PosixPath('/home/ethan/intel/openvino_notebooks/notebooks/117-model-server/models/detection/1/horizontal-text-detection-0001.bin')
Step 3: Start the Model Server Container
----------------------------------------
Pull and start the container:
Searching for an available serving port in local.
.. code:: ipython3
!docker run -d --rm --name="ovms" -v $(pwd)/models:/models -p 9000:9000 openvino/model_server:latest --model_path /models/detection/ --model_name detection --port 9000
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 0))
sock.listen(1)
port = sock.getsockname()[1]
sock.close()
print(f"Port {port} is available")
os.environ['port'] = str(port)
.. parsed-literal::
7bf50596c18d5ad93d131eb9e435439dfb3cedf994518c5e89cc7727f5d3530e
Port 39801 is available
.. code:: ipython3
!docker run -d --rm --name="ovms" -v $(pwd)/models:/models -p $port:9000 openvino/model_server:latest --model_path /models/detection/ --model_name detection --port 9000
.. parsed-literal::
64aa9391ba019b3ef26ae3010e5605e38d0a12e3f93bf74b3afb938f39b86ad2
Check whether the OVMS container is running normally:
@ -226,12 +264,12 @@ Check whether the OVMS container is running normally:
.. parsed-literal::
7bf50596c18d openvino/model_server:latest "/ovms/bin/ovms --mo…" Less than a second ago Up Less than a second 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp ovms
64aa9391ba01 openvino/model_server:latest "/ovms/bin/ovms --mo…" 29 seconds ago Up 28 seconds 0.0.0.0:37581->9000/tcp, :::37581->9000/tcp ovms
The required Model Server parameters are listed below. For additional
configuration options, see the `Model Server Parameters
section <https://docs.openvino.ai/2023.3/ovms_docs_parameters.html#doxid-ovms-docs-parameters>`__.
section <https://docs.openvino.ai/2023.2/ovms_docs_parameters.html>`__.
.. raw:: html
@ -648,22 +686,24 @@ openvino/model_server:latest
</table>
If the serving port ``9000`` is already in use, please switch it to
another available port on your system. For example:\ ``-p 9020:9000``
If the serving port is already in use, please switch it to another
available port on your system. For example:\ ``-p 9020:9000``
Step 4: Prepare the Example Client Components
---------------------------------------------------------------------------------------
Step 4: Prepare the Example Client Components
---------------------------------------------
OpenVINO Model Server exposes
two sets of APIs: one compatible with ``TensorFlow Serving`` and another
one, with ``KServe API``, for inference. Both APIs work on ``gRPC`` and
``REST``\ interfaces. Supporting two sets of APIs makes OpenVINO Model
Server easier to plug into existing systems the already leverage one of
these APIs for inference. This example will demonstrate how to write a
TensorFlow Serving API client for object detection.
Prerequisites
~~~~~~~~~~~~~
OpenVINO Model Server exposes two sets of APIs: one compatible with
``TensorFlow Serving`` and another one, with ``KServe API``, for
inference. Both APIs work on ``gRPC`` and ``REST``\ interfaces.
Supporting two sets of APIs makes OpenVINO Model Server easier to plug
into existing systems the already leverage one of these APIs for
inference. This example will demonstrate how to write a TensorFlow
Serving API client for object detection.
Prerequisites
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Install necessary packages.
@ -674,31 +714,13 @@ Install necessary packages.
.. parsed-literal::
Collecting ovmsclient
Downloading ovmsclient-2022.3-py3-none-any.whl (163 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.0/164.0 KB 2.1 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: numpy>=1.16.6 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (1.23.4)
Requirement already satisfied: requests>=2.27.1 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (2.27.1)
Collecting grpcio>=1.47.0
Downloading grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 5.6 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: protobuf>=3.19.4 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (3.19.6)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (1.26.9)
Requirement already satisfied: idna<4,>=2.5 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (2.0.12)
Installing collected packages: grpcio, ovmsclient
Attempting uninstall: grpcio
Found existing installation: grpcio 1.34.1
Uninstalling grpcio-1.34.1:
Successfully uninstalled grpcio-1.34.1
Successfully installed grpcio-1.51.3 ovmsclient-2022.3
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/home/adrian/repos/openvino_notebooks_adrian/venv/bin/python -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.
Imports
~~~~~~~
Imports
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -707,13 +729,15 @@ Imports
import matplotlib.pyplot as plt
from ovmsclient import make_grpc_client
Request Model Status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Request Model Status
~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
address = "localhost:9000"
address = "localhost:" + str(port)
# Bind the grpc address to the client object
client = make_grpc_client(address)
model_status = client.get_model_status(model_name=model_name)
@ -725,8 +749,10 @@ Request Model Status
{1: {'state': 'AVAILABLE', 'error_code': 0, 'error_message': 'OK'}}
Request Model Metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Request Model Metadata
~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -736,11 +762,13 @@ Request Model Metadata
.. parsed-literal::
{'model_version': 1, 'inputs': {'image': {'shape': [1, 3, 704, 704], 'dtype': 'DT_FLOAT'}}, 'outputs': {'1469_1470.0': {'shape': [-1], 'dtype': 'DT_FLOAT'}, '1078_1079.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1330_1331.0': {'shape': [36], 'dtype': 'DT_FLOAT'}, 'labels': {'shape': [-1], 'dtype': 'DT_INT32'}, '1267_1268.0': {'shape': [121], 'dtype': 'DT_FLOAT'}, '1141_1142.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1204_1205.0': {'shape': [484], 'dtype': 'DT_FLOAT'}, 'boxes': {'shape': [-1, 5], 'dtype': 'DT_FLOAT'}}}
{'model_version': 1, 'inputs': {'image': {'shape': [1, 3, 704, 704], 'dtype': 'DT_FLOAT'}}, 'outputs': {'boxes': {'shape': [-1, 5], 'dtype': 'DT_FLOAT'}, 'labels': {'shape': [-1], 'dtype': 'DT_INT64'}}}
Load input image
~~~~~~~~~~~~~~~~
Load input image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -749,43 +777,51 @@ Load input image
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/intel_rnb.jpg",
directory="data"
)
# Text detection models expect an image in BGR format.
image = cv2.imread(str(image_filename))
fp_image = image.astype("float32")
# Resize the image to meet network expected input sizes.
input_shape = model_metadata['inputs']['image']['shape']
height, width = input_shape[2], input_shape[3]
resized_image = cv2.resize(fp_image, (height, width))
# Reshape to the network input shape.
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
.. parsed-literal::
data/intel_rnb.jpg: 0%| | 0.00/288k [00:00<?, ?B/s]
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fee22d6ecd0>
<matplotlib.image.AxesImage at 0x7f254faeec50>
.. image:: 117-model-server-with-output_files/117-model-server-with-output_21_1.png
.. image:: 117-model-server-with-output_files/117-model-server-with-output_23_2.png
Request Prediction on a Numpy Array
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Request Prediction on a Numpy Array
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
inputs = {"image": input_image}
# Run inference on model server and receive the result data
boxes = client.predict(inputs=inputs, model_name=model_name)['boxes']
# Remove zero only boxes.
boxes = boxes[~np.all(boxes == 0, axis=1)]
print(boxes)
@ -793,16 +829,18 @@ Request Prediction on a Numpy Array
.. parsed-literal::
[[3.9992419e+02 8.1032524e+01 5.6187299e+02 1.3619952e+02 5.3706491e-01]
[2.6189725e+02 6.8310547e+01 3.8541251e+02 1.2095630e+02 4.7559953e-01]
[6.1644586e+02 2.8008759e+02 6.6627545e+02 3.1178854e+02 4.4982004e-01]
[2.0762042e+02 6.2798470e+01 2.3444728e+02 1.0706525e+02 3.7216505e-01]
[5.1742780e+02 5.5603595e+02 5.4927539e+02 5.8736023e+02 3.2588077e-01]
[2.2261986e+01 4.5406548e+01 1.8868817e+02 1.0225631e+02 3.0407205e-01]]
[[4.0075238e+02 8.1240105e+01 5.6262683e+02 1.3609659e+02 5.3646392e-01]
[2.6150497e+02 6.8225861e+01 3.8433078e+02 1.2111545e+02 4.7504124e-01]
[6.1611401e+02 2.8000638e+02 6.6605963e+02 3.1116574e+02 4.5030469e-01]
[2.0762566e+02 6.2619057e+01 2.3446707e+02 1.0711832e+02 3.7426147e-01]
[5.1753296e+02 5.5611102e+02 5.4918005e+02 5.8740009e+02 3.2477754e-01]
[2.2038467e+01 4.5390991e+01 1.8856328e+02 1.0215196e+02 2.9959568e-01]]
Visualization
~~~~~~~~~~~~~
Visualization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
@ -811,31 +849,31 @@ Visualization
def convert_result_to_image(bgr_image, resized_image, boxes, threshold=0.3, conf_labels=True):
# Define colors for boxes and descriptions.
colors = {"red": (255, 0, 0), "green": (0, 255, 0)}
# Fetch the image shapes to calculate a ratio.
(real_y, real_x), (resized_y, resized_x) = bgr_image.shape[:2], resized_image.shape[:2]
ratio_x, ratio_y = real_x / resized_x, real_y / resized_y
# Convert the base image from BGR to RGB format.
rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
# Iterate through non-zero boxes.
for box in boxes:
# Pick a confidence factor from the last place in an array.
conf = box[-1]
if conf > threshold:
# Convert float to int and multiply corner position of each box by x and y ratio.
# If the bounding box is found at the top of the image,
# position the upper box bar little lower to make it visible on the image.
# If the bounding box is found at the top of the image,
# position the upper box bar little lower to make it visible on the image.
(x_min, y_min, x_max, y_max) = [
int(max(corner_position * ratio_y, 10)) if idx % 2
int(max(corner_position * ratio_y, 10)) if idx % 2
else int(corner_position * ratio_x)
for idx, corner_position in enumerate(box[:-1])
]
# Draw a box based on the position, parameters in rectangle function are: image, start_point, end_point, color, thickness.
rgb_image = cv2.rectangle(rgb_image, (x_min, y_min), (x_max, y_max), colors["green"], 3)
# Add text to the image based on position and confidence.
# Parameters in text function are: image, text, bottom-left_corner_textfield, font, font_scale, color, thickness, line_type.
if conf_labels:
@ -849,7 +887,7 @@ Visualization
1,
cv2.LINE_AA,
)
return rgb_image
.. code:: ipython3
@ -863,12 +901,12 @@ Visualization
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fee219e4df0>
<matplotlib.image.AxesImage at 0x7f25490829b0>
.. image:: 117-model-server-with-output_files/117-model-server-with-output_26_1.png
.. image:: 117-model-server-with-output_files/117-model-server-with-output_28_1.png
To stop and remove the model server container, you can use the following
@ -884,10 +922,12 @@ command:
ovms
References
----------------------------------------------------
References
----------
1. `OpenVINO™ Model Server
documentation <https://docs.openvino.ai/2023.3/ovms_what_is_openvino_model_server.html>`__
documentation <https://docs.openvino.ai/2023.0/ovms_what_is_openvino_model_server.html>`__
2. `OpenVINO™ Model Server GitHub
repository <https://github.com/openvinotoolkit/model_server/>`__

3
docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_21_1.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e20e0cc48768c8a971c0e0cd72666c63f0bd5940256a1d43b0a2de2f7874eeef
size 112408

3
docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_23_2.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:151f6c5d9e90a68411adce7359011fe6580a071decbc7f1cacbf60975083bc9d
size 305482

3
docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_26_1.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:df4dbdbd56ef78ad7619bfb7fcfca6c96dccce598b3a5263486387b186e8d08f
size 232667

3
docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_28_1.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c0dd02ecae6935c5d65ae1ad543ef6a918185c440b667e68b98f4bd681762030
size 458294

8
docs/notebooks/117-model-server-with-output_files/index.html
View File

@ -1,8 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/117-model-server-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/117-model-server-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="117-model-server-with-output_21_1.png">117-model-server-with-output_21_1.png</a> 31-Oct-2023 00:35 112408
<a href="117-model-server-with-output_26_1.png">117-model-server-with-output_26_1.png</a> 31-Oct-2023 00:35 232667
</pre><hr></body>
</html>

8
docs/notebooks/117-model-server-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/117-model-server-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/117-model-server-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="117-model-server-with-output_23_2.png">117-model-server-with-output_23_2.png</a> 26-Jan-2024 01:04 305482
<a href="117-model-server-with-output_28_1.png">117-model-server-with-output_28_1.png</a> 26-Jan-2024 01:04 458294
</pre><hr></body>
</html>

51
docs/notebooks/118-optimize-preprocessing-with-output.rst
View File

@ -23,7 +23,8 @@ This tutorial include following steps:
- Comparing results on one picture.
- Comparing performance.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Settings <#settings>`__
- `Imports <#imports>`__
@ -39,7 +40,7 @@ This tutorial include following steps:
- `Convert model to OpenVINO IR with model conversion
API <#convert-model-to-openvino-ir-with-model-conversion-api>`__
- `Create PrePostProcessor
- `Create ``PrePostProcessor``
Object <#create-prepostprocessor-object>`__
- `Declare User’s Data Format <#declare-users-data-format>`__
- `Declaring Model Layout <#declaring-model-layout>`__
@ -105,10 +106,14 @@ Imports
.. parsed-literal::
2023-11-14 23:00:32.637266: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-14 23:00:32.671311: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-01-25 23:07:18.341652: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-01-25 23:07:18.375926: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-14 23:00:33.179278: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-01-25 23:07:18.888977: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Setup image and device
@ -189,12 +194,12 @@ and save it to the disk.
.. parsed-literal::
2023-11-14 23:00:37.345835: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2023-11-14 23:00:37.345869: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2023-11-14 23:00:37.345874: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2023-11-14 23:00:37.346012: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2023-11-14 23:00:37.346027: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2023-11-14 23:00:37.346030: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
2024-01-25 23:07:22.553946: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2024-01-25 23:07:22.553985: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2024-01-25 23:07:22.553989: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2024-01-25 23:07:22.554129: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2024-01-25 23:07:22.554144: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2024-01-25 23:07:22.554147: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
.. parsed-literal::
@ -268,7 +273,7 @@ Graph modifications of a model shall be performed after the model is
read from a drive and before it is loaded on the actual device.
Pre-processing support following operations (please, see more details
`here <https://docs.openvino.ai/2023.3/classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1aeacaf406d72a238e31a359798ebdb3b7>`__)
`here <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ov__dev__exec__model.html#_CPPv3N2ov10preprocess15PreProcessStepsE>`__)
- Mean/Scale Normalization
- Converting Precision
@ -304,7 +309,7 @@ Create ``PrePostProcessor`` Object
The
`PrePostProcessor() <https://docs.openvino.ai/2023.3/classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>`__
`PrePostProcessor() <https://docs.openvino.ai/2023.3/api/c_cpp_api/classov_1_1preprocess_1_1_pre_post_processor.html>`__
class enables specifying the preprocessing and postprocessing steps for
a model.
@ -329,7 +334,7 @@ about user’s input tensor will be initialized to same data
(type/shape/etc) as model’s input parameter. User application can
override particular parameters according to application’s data. Refer to
the following
`page <https://docs.openvino.ai/2023.3/classov_1_1preprocess_1_1InputTensorInfo.html#doxid-classov-1-1preprocess-1-1-input-tensor-info-1a98fb73ff9178c8c71d809ddf8927faf5>`__
`page <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ov__dev__exec__model.html#_CPPv4N2ov10preprocess15InputTensorInfoE>`__
for more information about parameters for overriding.
Below is all the specified input information:
@ -355,7 +360,7 @@ for mean/scale normalization.
.. parsed-literal::
<openvino._pyopenvino.preprocess.InputTensorInfo at 0x7fbffd787d70>
<openvino._pyopenvino.preprocess.InputTensorInfo at 0x7f5ce40abdb0>
@ -367,7 +372,7 @@ Declaring Model Layout
Model input already has information about precision and shape.
Preprocessing API is not intended to modify this. The only thing that
may be specified is input data
`layout <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Layout_Overview.html#doxid-openvino-docs-o-v-u-g-layout-overview>`__.
`layout <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Layout_Overview.html>`__.
.. code:: ipython3
@ -386,7 +391,7 @@ may be specified is input data
.. parsed-literal::
<openvino._pyopenvino.preprocess.InputModelInfo at 0x7fbffd7870b0>
<openvino._pyopenvino.preprocess.InputModelInfo at 0x7f5ce40abb70>
@ -406,7 +411,7 @@ Perform the following:
dynamic size, for example, ``{?, 3, ?, ?}`` resize will not know how
to resize the picture. Therefore, in this case, target height/ width
should be specified. For more details, see also the
`PreProcessSteps.resize() <https://docs.openvino.ai/2023.3/classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1a40dab78be1222fee505ed6a13400efe6>`__.
`PreProcessSteps.resize() <https://docs.openvino.ai/2023.3/api/ie_python_api/_autosummary/openvino.preprocess.PreProcessSteps.html#openvino.preprocess.PreProcessSteps.resize>`__.
- Subtract mean from each channel.
- Divide each pixel data to appropriate scale value.
@ -427,7 +432,7 @@ then such conversion will be added explicitly.
.. parsed-literal::
<openvino._pyopenvino.preprocess.PreProcessSteps at 0x7fc0a02556b0>
<openvino._pyopenvino.preprocess.PreProcessSteps at 0x7f5ce40abe70>
@ -646,6 +651,10 @@ Compare performance
.. parsed-literal::
IR model in OpenVINO Runtime/CPU with manual image preprocessing: 0.0152 seconds per image, FPS: 65.58
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0187 seconds per image, FPS: 53.52
IR model in OpenVINO Runtime/CPU with manual image preprocessing: 0.0149 seconds per image, FPS: 67.09
.. parsed-literal::
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0182 seconds per image, FPS: 54.89

2
docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_14_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7511b8a4e5b047600d5fed14fbc7e9653a868bc5253abf1e0c3ef649b47bc408
oid sha256:2dd4338c6c163e7693885ce544e8c9cd2aecedf3b136fa295e22877f37b5634c
size 387941

7
docs/notebooks/118-optimize-preprocessing-with-output_files/index.html
View File

@ -1,7 +0,0 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/118-optimize-preprocessing-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/118-optimize-preprocessing-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="118-optimize-preprocessing-with-output_14_1.png">118-optimize-preprocessing-with-output_14_1.png</a> 15-Nov-2023 00:43 387941
</pre><hr></body>
</html>

7
docs/notebooks/118-optimize-preprocessing-with-output_files/index.html.tmp Normal file
View File

@ -0,0 +1,7 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/118-optimize-preprocessing-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/118-optimize-preprocessing-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="118-optimize-preprocessing-with-output_14_1.png">118-optimize-preprocessing-with-output_14_1.png</a> 26-Jan-2024 01:04 387941
</pre><hr></body>
</html>

80
docs/notebooks/119-tflite-to-openvino-with-output.rst
View File

@ -14,8 +14,8 @@ IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Preparation <#preparation>`__
@ -47,7 +47,7 @@ Install requirements
%pip install -q "openvino>=2023.1.0"
%pip install -q opencv-python requests tqdm
# Fetch `notebook_utils` module
import urllib.request
urllib.request.urlretrieve(
@ -59,6 +59,10 @@ Install requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -73,7 +77,7 @@ Imports
import numpy as np
from PIL import Image
import openvino as ov
from notebook_utils import download_file, load_image
Download TFLite model
@ -85,10 +89,10 @@ Download TFLite model
model_dir = Path("model")
tflite_model_path = model_dir / "efficientnet_lite0_fp32_2.tflite"
ov_model_path = tflite_model_path.with_suffix(".xml")
model_url = "https://www.kaggle.com/models/tensorflow/efficientnet/frameworks/tfLite/variations/lite0-fp32/versions/2?lite-format=tflite"
download_file(model_url, tflite_model_path.name, model_dir)
@ -102,7 +106,7 @@ Download TFLite model
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite')
@ -122,7 +126,7 @@ serialization by ``ov.save_model``. For more information about model
conversion, see this
`page <https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html>`__.
For TensorFlow Lite models support, refer to this
`tutorial <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite.html>`__.
`tutorial <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite.html>`__.
.. code:: ipython3
@ -144,12 +148,12 @@ Load model using OpenVINO TensorFlow Lite Frontend
TensorFlow Lite models are supported via ``FrontEnd`` API. You may skip
conversion to IR and read models directly by OpenVINO runtime API. For
more examples supported formats reading via Frontend API, please look
this `tutorial <../002-openvino-api>`__.
this `tutorial <002-openvino-api-with-output.html>`__.
.. code:: ipython3
core = ov.Core()
ov_model = core.read_model(tflite_model_path)
Run OpenVINO model inference
@ -179,14 +183,14 @@ select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
@ -207,11 +211,11 @@ select device from dropdown list for running inference using OpenVINO
imagenet_classes_file_path = download_file("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/datasets/imagenet/imagenet_2012.txt")
imagenet_classes = open(imagenet_classes_file_path).read().splitlines()
top1_predicted_cls_id = np.argmax(predicted_scores)
top1_predicted_score = predicted_scores[0][top1_predicted_cls_id]
predicted_label = imagenet_classes[top1_predicted_cls_id]
display(image.resize((640, 512)))
print(f"Predicted label: {predicted_label} with probability {top1_predicted_score :2f}")
@ -235,7 +239,7 @@ Estimate Model Performance
--------------------------
`Benchmark
Tool <https://docs.openvino.ai/latest/openvino_sample_benchmark_tool.html>`__
Tool <https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html>`__
is used to measure the inference performance of the model on CPU and
GPU.
@ -258,22 +262,26 @@ GPU.
.. parsed-literal::
Benchmark model inference on CPU
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 28.35 ms
[ INFO ] Read model took 10.06 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] images (node: images) : f32 / [...] / [1,224,224,3]
@ -287,7 +295,11 @@ GPU.
[ INFO ] Model outputs:
[ INFO ] Softmax (node: 63) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 147.48 ms
.. parsed-literal::
[ INFO ] Compile model took 138.70 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: TensorFlow_Lite_Frontend_IR
@ -295,9 +307,9 @@ GPU.
[ INFO ] NUM_STREAMS: 6
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 24
[ INFO ] PERF_COUNT: False
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
@ -308,18 +320,22 @@ GPU.
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!
[ INFO ] Fill input 'images' with random values
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.23 ms
[ INFO ] First inference took 7.39 ms
.. parsed-literal::
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 17526 iterations
[ INFO ] Duration: 15005.83 ms
[ INFO ] Count: 17544 iterations
[ INFO ] Duration: 15006.83 ms
[ INFO ] Latency:
[ INFO ] Median: 5.00 ms
[ INFO ] Median: 5.01 ms
[ INFO ] Average: 5.00 ms
[ INFO ] Min: 2.72 ms
[ INFO ] Max: 15.43 ms
[ INFO ] Throughput: 1167.95 FPS
[ INFO ] Min: 2.90 ms
[ INFO ] Max: 14.27 ms
[ INFO ] Throughput: 1169.07 FPS

8
docs/notebooks/119-tflite-to-openvino-with-output_files/index.html → docs/notebooks/119-tflite-to-openvino-with-output_files/index.html.tmp
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/119-tflite-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/119-tflite-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/119-tflite-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="119-tflite-to-openvino-with-output_16_1.jpg">119-tflite-to-openvino-with-output_16_1.jpg</a> 07-Dec-2023 00:49 68170
<a href="119-tflite-to-openvino-with-output_16_1.png">119-tflite-to-openvino-with-output_16_1.png</a> 07-Dec-2023 00:49 621006
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/119-tflite-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="119-tflite-to-openvino-with-output_16_1.jpg">119-tflite-to-openvino-with-output_16_1.jpg</a> 26-Jan-2024 01:05 68170
<a href="119-tflite-to-openvino-with-output_16_1.png">119-tflite-to-openvino-with-output_16_1.png</a> 26-Jan-2024 01:05 621006
</pre><hr></body>
</html>

96
docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output.rst
View File

@ -18,11 +18,11 @@ Inception ResNet
V2 <https://tfhub.dev/tensorflow/mask_rcnn/inception_resnet_v2_1024x1024/1>`__
instance segmentation model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using `Model
Optimizer <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html>`__.
(OpenVINO IR) format, using `Model Conversion
API <https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html>`__.
After creating the OpenVINO IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
and do inference with a sample image.
**Table of contents:**
@ -66,7 +66,7 @@ The notebook uses utility functions. The cell below will download the
# Fetch the notebook utils script from the openvino_notebooks repo
import urllib.request
urllib.request.urlretrieve(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py",
filename="notebook_utils.py",
@ -81,15 +81,15 @@ Imports
# Standard python modules
from pathlib import Path
# External modules and dependencies
import cv2
import matplotlib.pyplot as plt
import numpy as np
# Notebook utils module
from notebook_utils import download_file
# OpenVINO modules
import openvino as ov
@ -105,21 +105,21 @@ Define model related variables and create corresponding directories:
# Create directories for models files
model_dir = Path("model")
model_dir.mkdir(exist_ok=True)
# Create directory for TensorFlow model
tf_model_dir = model_dir / "tf"
tf_model_dir.mkdir(exist_ok=True)
# Create directory for OpenVINO IR model
ir_model_dir = model_dir / "ir"
ir_model_dir.mkdir(exist_ok=True)
model_name = "mask_rcnn_inception_resnet_v2_1024x1024"
openvino_ir_path = ir_model_dir / f"{model_name}.xml"
tf_model_url = "https://www.kaggle.com/models/tensorflow/mask-rcnn-inception-resnet-v2/frameworks/tensorFlow2/variations/1024x1024/versions/1?tf-hub-format=compressed"
tf_model_archive_filename = f"{model_name}.tar.gz"
Download Model from TensorFlow Hub
@ -152,7 +152,7 @@ archive:
.. code:: ipython3
import tarfile
with tarfile.open(tf_model_dir / tf_model_archive_filename) as file:
file.extractall(path=tf_model_dir)
@ -180,7 +180,7 @@ when the model is run in the future.
.. code:: ipython3
ov_model = ov.convert_model(tf_model_dir)
# Save converted OpenVINO IR model to the corresponding directory
ov.save_model(ov_model, openvino_ir_path)
@ -199,7 +199,7 @@ select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
@ -207,7 +207,7 @@ select device from dropdown list for running inference using OpenVINO
description='Device:',
disabled=False,
)
device
@ -263,12 +263,12 @@ the first (and highest) detection score.
model_inputs = compiled_model.inputs
model_outputs = compiled_model.outputs
print("Model inputs count:", len(model_inputs))
print("Model inputs:")
for _input in model_inputs:
print(" ", _input)
print("Model outputs count:", len(model_outputs))
print("Model outputs:")
for output in model_outputs:
@ -317,7 +317,7 @@ Load and save an image:
.. code:: ipython3
image_path = Path("./data/coco_bike.jpg")
download_file(
url="https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg",
filename=image_path.name,
@ -337,16 +337,16 @@ Read the image, resize and convert it to the input shape of the network:
# Read the image
image = cv2.imread(filename=str(image_path))
# The network expects images in RGB format
image = cv2.cvtColor(image, code=cv2.COLOR_BGR2RGB)
# Resize the image to the network input shape
resized_image = cv2.resize(src=image, dsize=(255, 255))
# Add batch dimension to image
network_input_image = np.expand_dims(resized_image, 0)
# Show the image
plt.imshow(image)
@ -355,7 +355,7 @@ Read the image, resize and convert it to the input shape of the network:
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f53cb91bca0>
<matplotlib.image.AxesImage at 0x7f5f2fe7a310>
@ -382,23 +382,23 @@ be extracted from the result. For further model result visualization
detection_boxes = compiled_model.output("detection_boxes")
image_detection_boxes = inference_result[detection_boxes]
print("image_detection_boxes:", image_detection_boxes.shape)
detection_masks = compiled_model.output("detection_masks")
image_detection_masks = inference_result[detection_masks]
print("image_detection_masks:", image_detection_masks.shape)
detection_classes = compiled_model.output("detection_classes")
image_detection_classes = inference_result[detection_classes]
print("image_detection_classes:", image_detection_classes.shape)
detection_scores = compiled_model.output("detection_scores")
image_detection_scores = inference_result[detection_scores]
print("image_detection_scores:", image_detection_scores.shape)
num_detections = compiled_model.output("num_detections")
image_num_detections = inference_result[num_detections]
print("image_detections_num:", image_num_detections)
# Alternatively, inference result data can be extracted by model output name with `.get()` method
assert (inference_result[detection_boxes] == inference_result.get("detection_boxes")).all(), "extracted inference result data should be equal"
@ -423,14 +423,14 @@ Define utility functions to visualize the inference results
import random
from typing import Optional
def add_detection_box(
box: np.ndarray, image: np.ndarray, mask: np.ndarray, label: Optional[str] = None
) -> np.ndarray:
"""
Helper function for adding single bounding box to the image
Parameters
----------
box : np.ndarray
@ -441,18 +441,18 @@ Define utility functions to visualize the inference results
Segmentation mask in format (H, W)
label : str, optional
Detection box label string, if not provided will not be added to result image (default is None)
Returns
-------
np.ndarray
NumPy array including image, detection box, and segmentation mask
"""
ymin, xmin, ymax, xmax = box
point1, point2 = (int(xmin), int(ymin)), (int(xmax), int(ymax))
box_color = [random.randint(0, 255) for _ in range(3)]
line_thickness = round(0.002 * (image.shape[0] + image.shape[1]) / 2) + 1
result = cv2.rectangle(
img=image,
pt1=point1,
@ -461,7 +461,7 @@ Define utility functions to visualize the inference results
thickness=line_thickness,
lineType=cv2.LINE_AA,
)
if label:
font_thickness = max(line_thickness - 1, 1)
font_face = 0
@ -504,12 +504,12 @@ Define utility functions to visualize the inference results
def get_mask_frame(box, frame, mask):
"""
Transform a binary mask to fit within a specified bounding box in a frame using perspective transformation.
Args:
box (tuple): A bounding box represented as a tuple (y_min, x_min, y_max, x_max).
frame (numpy.ndarray): The larger frame or image where the mask will be placed.
mask (numpy.ndarray): A binary mask image to be transformed.
Returns:
numpy.ndarray: A transformed mask image that fits within the specified bounding box in the frame.
"""
@ -534,10 +534,10 @@ Define utility functions to visualize the inference results
.. code:: ipython3
from typing import Dict
from openvino.runtime.utils.data_helpers import OVDict
def visualize_inference_result(
inference_result: OVDict,
image: np.ndarray,
@ -546,7 +546,7 @@ Define utility functions to visualize the inference results
):
"""
Helper function for visualizing inference result on the image
Parameters
----------
inference_result : OVDict
@ -563,13 +563,13 @@ Define utility functions to visualize the inference results
detection_scores = inference_result.get("detection_scores")
num_detections = inference_result.get("num_detections")
detection_masks = inference_result.get("detection_masks")
detections_limit = int(
min(detections_limit, num_detections[0])
if detections_limit is not None
else num_detections[0]
)
# Normalize detection boxes coordinates to original image size
original_image_height, original_image_width, _ = image.shape
normalized_detection_boxes = detection_boxes[0, :detections_limit] * [
@ -589,7 +589,7 @@ Define utility functions to visualize the inference results
result = add_detection_box(
box=normalized_detection_boxes[i], image=result, mask=mask_reframed, label=label
)
plt.imshow(result)
TensorFlow Instance Segmentation model
@ -605,7 +605,7 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__:
.. code:: ipython3
coco_labels_file_path = Path("./data/coco_91cl.txt")
download_file(
url="https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/coco_91cl.txt",
filename=coco_labels_file_path.name,
@ -628,7 +628,7 @@ file:
with open(coco_labels_file_path, "r") as file:
coco_labels = file.read().strip().split("\n")
coco_labels_map = dict(enumerate(coco_labels, 1))
print(coco_labels_map)

4
docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1359365695e66164137690b56f261eae3364dff68f3e155a3e6b878ff17cea3f
size 393190
oid sha256:ed008919b91d0e231d8872fb4fae77bdfc3a513547568b9448b69de0065625c5
size 393453

8
docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html → docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html.tmp
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-instance-segmentation-to-openvino-with-output_25_1.png">120-tensorflow-instance-segmentation-to-openvin..&gt;</a> 07-Dec-2023 00:49 395346
<a href="120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png">120-tensorflow-instance-segmentation-to-openvin..&gt;</a> 07-Dec-2023 00:49 393190
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-instance-segmentation-to-openvino-with-output_25_1.png">120-tensorflow-instance-segmentation-to-openvin..&gt;</a> 26-Jan-2024 01:05 395346
<a href="120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png">120-tensorflow-instance-segmentation-to-openvin..&gt;</a> 26-Jan-2024 01:05 393453
</pre><hr></body>
</html>

110
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst
View File

@ -20,11 +20,11 @@ object detection model to OpenVINO `Intermediate
Representation <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_IR_and_opsets.html>`__
(OpenVINO IR) format, using Model Converter. After creating the OpenVINO
IR, load the model in `OpenVINO
Runtime <https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
Runtime <https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_OV_Runtime_User_Guide.html>`__
and do inference with a sample image.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Imports <#imports>`__
@ -74,7 +74,7 @@ The notebook uses utility functions. The cell below will download the
# Fetch the notebook utils script from the openvino_notebooks repo
import urllib.request
urllib.request.urlretrieve(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py",
filename="notebook_utils.py",
@ -89,14 +89,14 @@ Imports
# Standard python modules
from pathlib import Path
# External modules and dependencies
import cv2
import matplotlib.pyplot as plt
import numpy as np
# OpenVINO import
import openvino as ov
# Notebook utils module
from notebook_utils import download_file
@ -112,21 +112,21 @@ Define model related variables and create corresponding directories:
# Create directories for models files
model_dir = Path("model")
model_dir.mkdir(exist_ok=True)
# Create directory for TensorFlow model
tf_model_dir = model_dir / "tf"
tf_model_dir.mkdir(exist_ok=True)
# Create directory for OpenVINO IR model
ir_model_dir = model_dir / "ir"
ir_model_dir.mkdir(exist_ok=True)
model_name = "faster_rcnn_resnet50_v1_640x640"
openvino_ir_path = ir_model_dir / f"{model_name}.xml"
tf_model_url = "https://www.kaggle.com/models/tensorflow/faster-rcnn-resnet-v1/frameworks/tensorFlow2/variations/faster-rcnn-resnet50-v1-640x640/versions/1?tf-hub-format=compressed"
tf_model_archive_filename = f"{model_name}.tar.gz"
Download Model from TensorFlow Hub
@ -157,7 +157,7 @@ from TensorFlow Hub:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz')
@ -166,7 +166,7 @@ Extract TensorFlow Object Detection model from the downloaded archive:
.. code:: ipython3
import tarfile
with tarfile.open(tf_model_dir / tf_model_archive_filename) as file:
file.extractall(path=tf_model_dir)
@ -175,8 +175,8 @@ Convert Model to OpenVINO IR
OpenVINO Model Converter Python API can be used to convert the
TensorFlow model to OpenVINO IR.
OpenVINO Model Conversion API can be used to convert the TensorFlow
model to OpenVINO IR.
``ov.convert_model`` function accept path to TensorFlow model and
returns OpenVINO Model class instance which represents this model. Also
@ -188,15 +188,15 @@ The converted model is ready to load on a device using ``compile_model``
or saved on disk using the ``save_model`` function to reduce loading
time when the model is run in the future.
See the `Model Converter Developer
See the `Model Preparation
Guide <https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html>`__
for more information about Model Converter and TensorFlow `models
support <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
for more information about model conversion and TensorFlow `models
support <https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html>`__.
.. code:: ipython3
ov_model = ov.convert_model(tf_model_dir)
# Save converted OpenVINO IR model to the corresponding directory
ov.save_model(ov_model, openvino_ir_path)
@ -215,7 +215,7 @@ select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
@ -223,7 +223,7 @@ select device from dropdown list for running inference using OpenVINO
description='Device:',
disabled=False,
)
device
@ -290,10 +290,10 @@ for more information about model inputs, outputs and their formats.
model_inputs = compiled_model.inputs
model_input = compiled_model.input(0)
model_outputs = compiled_model.outputs
print("Model inputs count:", len(model_inputs))
print("Model input:", model_input)
print("Model outputs count:", len(model_outputs))
print("Model outputs:")
for output in model_outputs:
@ -326,7 +326,7 @@ Load and save an image:
.. code:: ipython3
image_path = Path("./data/coco_bike.jpg")
download_file(
url="https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg",
filename=image_path.name,
@ -343,7 +343,7 @@ Load and save an image:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg')
@ -353,16 +353,16 @@ Read the image, resize and convert it to the input shape of the network:
# Read the image
image = cv2.imread(filename=str(image_path))
# The network expects images in RGB format
image = cv2.cvtColor(image, code=cv2.COLOR_BGR2RGB)
# Resize the image to the network input shape
resized_image = cv2.resize(src=image, dsize=(255, 255))
# Transpose the image to the network input shape
network_input_image = np.expand_dims(resized_image, 0)
# Show the image
plt.imshow(image)
@ -371,7 +371,7 @@ Read the image, resize and convert it to the input shape of the network:
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7f74d41406d0>
<matplotlib.image.AxesImage at 0x7f8f0a2c3d00>
@ -396,19 +396,19 @@ outputs will be used.
.. code:: ipython3
_, detection_boxes, detection_classes, _, detection_scores, num_detections, _, _ = model_outputs
image_detection_boxes = inference_result[detection_boxes]
print("image_detection_boxes:", image_detection_boxes)
image_detection_classes = inference_result[detection_classes]
print("image_detection_classes:", image_detection_classes)
image_detection_scores = inference_result[detection_scores]
print("image_detection_scores:", image_detection_scores)
image_num_detections = inference_result[num_detections]
print("image_detections_num:", image_num_detections)
# Alternatively, inference result data can be extracted by model output name with `.get()` method
assert (inference_result[detection_boxes] == inference_result.get("detection_boxes")).all(), "extracted inference result data should be equal"
@ -503,12 +503,12 @@ Define utility functions to visualize the inference results
import random
from typing import Optional
def add_detection_box(box: np.ndarray, image: np.ndarray, label: Optional[str] = None) -> np.ndarray:
"""
Helper function for adding single bounding box to the image
Parameters
----------
box : np.ndarray
@ -517,20 +517,20 @@ Define utility functions to visualize the inference results
The image to which detection box is added
label : str, optional
Detection box label string, if not provided will not be added to result image (default is None)
Returns
-------
np.ndarray
NumPy array including both image and detection box
"""
ymin, xmin, ymax, xmax = box
point1, point2 = (int(xmin), int(ymin)), (int(xmax), int(ymax))
box_color = [random.randint(0, 255) for _ in range(3)]
line_thickness = round(0.002 * (image.shape[0] + image.shape[1]) / 2) + 1
cv2.rectangle(img=image, pt1=point1, pt2=point2, color=box_color, thickness=line_thickness, lineType=cv2.LINE_AA)
if label:
font_thickness = max(line_thickness - 1, 1)
font_face = 0
@ -551,14 +551,14 @@ Define utility functions to visualize the inference results
.. code:: ipython3
from typing import Dict
from openvino.runtime.utils.data_helpers import OVDict
def visualize_inference_result(inference_result: OVDict, image: np.ndarray, labels_map: Dict, detections_limit: Optional[int] = None):
"""
Helper function for visualizing inference result on the image
Parameters
----------
inference_result : OVDict
@ -574,13 +574,13 @@ Define utility functions to visualize the inference results
detection_classes: np.ndarray = inference_result.get("detection_classes")
detection_scores: np.ndarray = inference_result.get("detection_scores")
num_detections: np.ndarray = inference_result.get("num_detections")
detections_limit = int(
min(detections_limit, num_detections[0])
if detections_limit is not None
else num_detections[0]
)
# Normalize detection boxes coordinates to original image size
original_image_height, original_image_width, _ = image.shape
normalized_detection_boxex = detection_boxes[::] * [
@ -589,9 +589,9 @@ Define utility functions to visualize the inference results
original_image_height,
original_image_width,
]
image_with_detection_boxex = np.copy(image)
for i in range(detections_limit):
detected_class_name = labels_map[int(detection_classes[0, i])]
score = detection_scores[0, i]
@ -601,7 +601,7 @@ Define utility functions to visualize the inference results
image=image_with_detection_boxex,
label=label,
)
plt.imshow(image_with_detection_boxex)
TensorFlow Object Detection model
@ -617,7 +617,7 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__:
.. code:: ipython3
coco_labels_file_path = Path("./data/coco_91cl.txt")
download_file(
url="https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/coco_91cl.txt",
filename=coco_labels_file_path.name,
@ -635,7 +635,7 @@ Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__:
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt')
@ -648,7 +648,7 @@ file:
with open(coco_labels_file_path, "r") as file:
coco_labels = file.read().strip().split("\n")
coco_labels_map = dict(enumerate(coco_labels, 1))
print(coco_labels_map)

4
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ec79c39b72b7a139eca9f5b7081efec33a47fa8c55c9a459340774a045f2525b
size 391797
oid sha256:a5b7ad5bc51c70dfab0037ca885aeb4e613947df3fd7fdff8d055c8d64aefd4a
size 391815

8
docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html → docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html.tmp
View File

@ -1,8 +1,8 @@
<html>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</title></head>
<head><title>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</title></head>
<body bgcolor="white">
<h1>Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-object-detection-to-openvino-with-output_25_1.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 07-Dec-2023 00:49 395346
<a href="120-tensorflow-object-detection-to-openvino-with-output_38_0.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 07-Dec-2023 00:49 391797
<h1>Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/</h1><hr><pre><a href="../">../</a>
<a href="120-tensorflow-object-detection-to-openvino-with-output_25_1.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 26-Jan-2024 01:05 395346
<a href="120-tensorflow-object-detection-to-openvino-with-output_38_0.png">120-tensorflow-object-detection-to-openvino-wit..&gt;</a> 26-Jan-2024 01:05 391815
</pre><hr></body>
</html>

1471
docs/notebooks/121-convert-to-openvino-with-output.rst
View File

File diff suppressed because it is too large Load Diff

1769
docs/notebooks/121-legacy-mo-convert-to-openvino-with-output.rst Normal file
View File

File diff suppressed because it is too large Load Diff

160
docs/notebooks/122-speech-recognition-quantization-wav2vec2-with-output.rst
View File

File diff suppressed because one or more lines are too long

77
docs/notebooks/122-yolov8-quantization-with-accuracy-control-with-output.rst
View File

@ -38,8 +38,8 @@ and has the following differences:
The steps for the quantization with accuracy control are described
below.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Get Pytorch model and OpenVINO IR
@ -95,7 +95,7 @@ we do not need to do these steps manually.
import os
from pathlib import Path
from ultralytics import YOLO
from ultralytics.yolo.cfg import get_cfg
from ultralytics.yolo.data.utils import check_det_dataset
@ -104,11 +104,11 @@ we do not need to do these steps manually.
from ultralytics.yolo.utils import DEFAULT_CFG
from ultralytics.yolo.utils import ops
from ultralytics.yolo.utils.metrics import ConfusionMatrix
ROOT = os.path.abspath('')
MODEL_NAME = "yolov8n-seg"
model = YOLO(f"{ROOT}/{MODEL_NAME}.pt")
args = get_cfg(cfg=DEFAULT_CFG)
args.data = "coco128-seg.yaml"
@ -118,12 +118,12 @@ Load model.
.. code:: ipython3
import openvino as ov
model_path = Path(f"{ROOT}/{MODEL_NAME}_openvino_model/{MODEL_NAME}.xml")
if not model_path.exists():
model.export(format="openvino", dynamic=True, half=False)
ov_model = ov.Core().read_model(model_path)
Define validator and data loader
@ -146,7 +146,7 @@ validator class instance.
validator = model.ValidatorClass(args)
validator.data = check_det_dataset(args.data)
data_loader = validator.get_dataloader(f"{DATASETS_DIR}/coco128-seg", 1)
validator.is_coco = True
validator.class_map = ops.coco80_to_coco91_class()
validator.names = model.model.names
@ -173,15 +173,15 @@ We can use one dataset as calibration and validation datasets. Name it
.. code:: ipython3
from typing import Dict
import nncf
def transform_fn(data_item: Dict):
input_tensor = validator.preprocess(data_item)["img"].numpy()
return input_tensor
quantization_dataset = nncf.Dataset(data_loader, transform_fn)
@ -198,11 +198,11 @@ Prepare validation function
.. code:: ipython3
from functools import partial
import torch
from nncf.quantization.advanced_parameters import AdvancedAccuracyRestorerParameters
def validation_ac(
compiled_model: ov.CompiledModel,
validation_loader: torch.utils.data.DataLoader,
@ -216,7 +216,7 @@ Prepare validation function
validator.batch_i = 1
validator.confusion_matrix = ConfusionMatrix(nc=validator.nc)
num_outputs = len(compiled_model.outputs)
counter = 0
for batch_i, batch in enumerate(validation_loader):
if num_samples is not None and batch_i == num_samples:
@ -240,10 +240,10 @@ Prepare validation function
stats_metrics = stats["metrics/mAP50-95(M)"]
if log:
print(f"Validate: dataset length = {counter}, metric value = {stats_metrics:.3f}")
return stats_metrics
validation_fn = partial(validation_ac, validator=validator, log=False)
Run quantization with accuracy control
@ -297,6 +297,7 @@ value 25 to speed up the execution.
INFO:nncf:Validation of initial model was started
.. parsed-literal::
INFO:nncf:Elapsed Time: 00:00:00
@ -318,13 +319,13 @@ value 25 to speed up the execution.
INFO:nncf:Calculating ranking score for groups of quantizers
INFO:nncf:Elapsed Time: 00:02:16
INFO:nncf:Changing the scope of quantizer nodes was started
INFO:nncf:Reverted 1 operations to the floating-point precision:
INFO:nncf:Reverted 1 operations to the floating-point precision:
/model.22/Mul_5
INFO:nncf:Accuracy drop with the new quantization scope is 0.013359187935064742 (DropType.ABSOLUTE)
INFO:nncf:Reverted 1 operations to the floating-point precision:
INFO:nncf:Reverted 1 operations to the floating-point precision:
/model.1/conv/Conv/WithoutBiases
INFO:nncf:Accuracy drop with the new quantization scope is 0.01287864227202773 (DropType.ABSOLUTE)
INFO:nncf:Reverted 1 operations to the floating-point precision:
INFO:nncf:Reverted 1 operations to the floating-point precision:
/model.2/cv1/conv/Conv/WithoutBiases
INFO:nncf:Algorithm completed: achieved required accuracy drop 0.007027355074555763 (DropType.ABSOLUTE)
INFO:nncf:3 out of 91 were reverted back to the floating-point precision:
@ -347,11 +348,11 @@ is not exceeded.
core = ov.Core()
quantized_compiled_model = core.compile_model(model=quantized_model, device_name='CPU')
compiled_ov_model = core.compile_model(model=ov_model, device_name='CPU')
pt_result = validation_ac(compiled_ov_model, data_loader, validator)
quantized_result = validation_ac(quantized_compiled_model, data_loader, validator)
print(f'[Original OpenVino]: {pt_result:.4f}')
print(f'[Quantized OpenVino]: {quantized_result:.4f}')
@ -372,10 +373,10 @@ And compare performance.
# Set model directory
MODEL_DIR = Path("model")
MODEL_DIR.mkdir(exist_ok=True)
ir_model_path = MODEL_DIR / 'ir_model.xml'
quantized_model_path = MODEL_DIR / 'quantized_model.xml'
# Save models to use them in the commandline banchmark app
ov.save_model(ov_model, ir_model_path, compress_to_fp16=False)
ov.save_model(quantized_model, quantized_model_path, compress_to_fp16=False)
@ -393,12 +394,12 @@ And compare performance.
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-12713-47c2a91b6b6
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-12713-47c2a91b6b6
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
@ -442,7 +443,7 @@ And compare performance.
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!
[ INFO ] Fill input 'images' with random values
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 42.88 ms
@ -471,12 +472,12 @@ And compare performance.
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-12713-47c2a91b6b6
[ INFO ]
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-12713-47c2a91b6b6
[ INFO ]
[ INFO ]
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
@ -520,7 +521,7 @@ And compare performance.
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!
[ INFO ] Fill input 'images' with random values
[ INFO ] Fill input 'images' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 31.29 ms

65
docs/notebooks/123-detectron2-to-openvino-with-output.rst
View File

@ -1,4 +1,5 @@
# Convert Detectron2 Models to OpenVINO™
Convert Detectron2 Models to OpenVINO™
=========================================
`Detectron2 <https://github.com/facebookresearch/detectron2>`__ is
Facebook AI Research’s library that provides state-of-the-art detection
@ -14,8 +15,8 @@ using OpenVINO™. We will use ``Faster R-CNN FPN x1`` model and
`COCO <https://cocodataset.org/#home>`__ dataset as examples for object
detection and instance segmentation respectively.
**Table of contents:**
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
@ -59,7 +60,15 @@ Install required packages for running model
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -84,12 +93,12 @@ reading model config.
.. code:: ipython3
import detectron2.model_zoo as detectron_zoo
def get_model_and_config(model_name:str):
"""
Helper function for downloading PyTorch model and its configuration from Detectron2 Model Zoo
Parameters:
model_name (str): model_id from Detectron2 Model Zoo
Returns:
@ -122,12 +131,12 @@ simplify model’s structure making it more export-friendly.
import openvino as ov
import warnings
from typing import List, Dict
def convert_detectron2_model(model:torch.nn.Module, sample_input:List[Dict[str, torch.Tensor]]):
"""
Function for converting Detectron2 models, creates TracingAdapter for making model tracing-friendly,
prepares inputs and converts model to OpenVINO Model
Parameters:
model (torch.nn.Module): Model object for conversion
sample_input (List[Dict[str, torch.Tensor]]): sample input for tracing
@ -136,7 +145,7 @@ simplify model’s structure making it more export-friendly.
"""
# prepare input for tracing adapter
tracing_input = [{'image': sample_input[0]["image"]}]
# override model forward and disable postprocessing if required
if isinstance(model, GeneralizedRCNN):
def inference(model, inputs):
@ -145,7 +154,7 @@ simplify model’s structure making it more export-friendly.
return [{"instances": inst}]
else:
inference = None # assume that we just call the model directly
# create traceable model
traceable_model = TracingAdapter(model, tracing_input, inference)
warnings.filterwarnings("ignore")
@ -167,23 +176,23 @@ steps based on model specific transformations defined in model config.
import requests
from pathlib import Path
from PIL import Image
MODEL_DIR = Path("model")
DATA_DIR = Path("data")
MODEL_DIR.mkdir(exist_ok=True)
DATA_DIR.mkdir(exist_ok=True)
input_image_url = "https://farm9.staticflickr.com/8040/8017130856_1b46b5f5fc_z.jpg"
image_file = DATA_DIR / "example_image.jpg"
if not image_file.exists():
image = Image.open(requests.get(input_image_url, stream=True).raw)
image.save(image_file)
else:
image = Image.open(image_file)
image
@ -198,7 +207,7 @@ steps based on model specific transformations defined in model config.
import detectron2.data.transforms as T
from detectron2.data import detection_utils
import torch
def get_sample_inputs(image_path, cfg):
# get a sample data
original_image = detection_utils.read_image(image_path, format=cfg.INPUT.FORMAT)
@ -207,9 +216,9 @@ steps based on model specific transformations defined in model config.
height, width = original_image.shape[:2]
image = aug.get_transform(original_image).apply_image(original_image)
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))
inputs = {"image": image, "height": height, "width": width}
# Sample ready
sample_inputs = [inputs]
return sample_inputs
@ -264,16 +273,16 @@ select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
device
@ -325,11 +334,11 @@ provide helpers for wrapping output in original Detectron2 format.
from detectron2.utils.visualizer import ColorMode, Visualizer
from detectron2.data import MetadataCatalog
import numpy as np
def postprocess_detection_result(outputs:Dict, orig_height:int, orig_width:int, conf_threshold:float = 0.0):
"""
Helper function for postprocessing prediction results
Parameters:
outputs (Dict): OpenVINO model output dictionary
orig_height (int): original image height before preprocessing
@ -354,17 +363,17 @@ provide helpers for wrapping output in original Detectron2 format.
out_dict["pred_masks"] = torch.from_numpy(masks)
instances = Instances(model_input_size, **out_dict)
return detector_postprocess(instances, orig_height, orig_width)
def draw_instance_prediction(img:np.ndarray, results:Instances, cfg:"Config"):
"""
Helper function for visualization prediction results
Parameters:
img (np.ndarray): original image for drawing predictions
results (instances): model predictions
cfg (Config): model configuration
Returns:
img_with_res: image with results
img_with_res: image with results
"""
metadata = MetadataCatalog.get(cfg.DATASETS.TEST[0])
visualizer = Visualizer(img, metadata, instance_mode=ColorMode.IMAGE)
@ -414,7 +423,7 @@ Convert Instance Segmentation Model to OpenVINO Intermediate Representation
.. code:: ipython3
model_xml_path = MODEL_DIR / (model_name.split("/")[-1] + '.xml')
if not model_xml_path.exists():
ov_model = convert_detectron2_model(model, sample_input)
ov.save_model(ov_model, MODEL_DIR / (model_name.split("/")[-1] + '.xml'))

4
docs/notebooks/123-detectron2-to-openvino-with-output_files/123-detectron2-to-openvino-with-output_22_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d6f08e4b663ac8ea6d0970804111a5ffc4e7aac44f24b2a7121f15c9900fa535
size 59666
oid sha256:b9a59f5b6bc77d33212bcadd69faacc6c286d6eac7d7ed88ce28ccefe227f010
size 59043

4
docs/notebooks/123-detectron2-to-openvino-with-output_files/123-detectron2-to-openvino-with-output_22_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a4a06ea24c3a51aead9c90815b867910e7be07e8b8d443c89ad4003695dcf48d
size 508841
oid sha256:2db4d40d09643561d54ed996c0e42df721272e7bc17220c20240a81142df67eb
size 508506

4
docs/notebooks/123-detectron2-to-openvino-with-output_files/123-detectron2-to-openvino-with-output_32_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cacd033467ea9cdf6ea9db93bb70af3837dd492a870bac492683badc7643ed6d
size 55922
oid sha256:44dbb04db8f71ea61f1896b341ac6811246dd60d600942296da3918b7f82c715
size 55966

4
docs/notebooks/123-detectron2-to-openvino-with-output_files/123-detectron2-to-openvino-with-output_32_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0f1347884309a3631eb6eb2c63a44fa995c158b1e747c5230974eaada4fda4d0
size 459385
oid sha256:b9e00a023f05b67dd28e92c59ea377d5c9dc85f2c4f183d44e03227b25384749
size 457794

Some files were not shown because too many files have changed in this diff Show More