diff --git a/docs/articles_en/learn_openvino/tutorials.rst b/docs/articles_en/learn_openvino/tutorials.rst index ee0c83616f0..f3546334982 100644 --- a/docs/articles_en/learn_openvino/tutorials.rst +++ b/docs/articles_en/learn_openvino/tutorials.rst @@ -59,6 +59,45 @@ The Jupyter notebooks are categorized into following classes: Below you will find a selection of recommended tutorials that demonstrate inference on a particular model. These tutorials are guaranteed to provide a great experience with inference in OpenVINO: +.. showcase:: + :title: 280-depth-anything + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/280-depth-anything/280-depth-anything.gif + + Depth estimation with DepthAnything and OpenVINO. + +.. showcase:: + :title: 279-mobilevlm-language-assistant + :img: _static/images/notebook_eye.png + + Mobile language assistant with MobileVLM and OpenVINO. + +.. showcase:: + :title: 278-stable-diffusion-ip-adapter + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/278-stable-diffusion-ip-adapter/278-stable-diffusion-ip-adapter.png + + Image Generation with Stable Diffusion and IP-Adapter. + +.. showcase:: + :title: 275-llm-question-answering + :img: _static/images/notebook_eye.png + + LLM Instruction-following pipeline with OpenVINO. + + +.. showcase:: + :title: 274-efficient-sam + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/274-efficient-sam/274-efficient-sam.png + + Object segmentations with EfficientSAM and OpenVINO. + + +.. showcase:: + :title: 273-stable-zephyr-3b-chatbot + :img: _static/images/notebook_eye.png + + LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO. + + .. showcase:: :title: 272-paint-by-example :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/272-paint-by-example/272-paint-by-example.png @@ -83,41 +122,6 @@ Below you will find a selection of recommended tutorials that demonstrate infere Frame interpolation using FILM and OpenVINO. -.. showcase:: - :title: 267-distil-whisper-asr - :img: _static/images/notebook_eye.png - - Automatic speech recognition using Distil-Whisper and OpenVINO. - -.. showcase:: - :title: 265-wuerstchen-image-generation - :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/265-wuerstchen-image-generation/265-wuerstchen-image-generation.png - - Image generation with Würstchen and OpenVINO. - -.. showcase:: - :title: 264-qrcode-monster - :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/264-qrcode-monster/264-qrcode-monster.png - - Generate creative QR codes with ControlNet QR Code Monster and OpenVINO. - -.. showcase:: - :title: 263-latent-consistency-models-image-generation - :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/263-latent-consistency-models-image-generation/263-latent-consistency-models-image-generation.png - - Image generation with Latent Consistency Model and OpenVINO. - -.. showcase:: - :title: 263-lcm-lora-controlnet - :img: https://user-images.githubusercontent.com/29454499/284292122-f146e16d-7233-49f7-a401-edcb714b5288.png - - Text-to-Image Generation with LCM LoRA and ControlNet Conditioning. - -.. showcase:: - :title: 262-softvc-voice-conversion - :img: _static/images/notebook_eye.png - - SoftVC VITS Singing Voice Conversion and OpenVINO. .. note:: diff --git a/docs/articles_en/learn_openvino/tutorials/notebooks_section_1.rst b/docs/articles_en/learn_openvino/tutorials/notebooks_section_1.rst index 767f1661153..67906f50679 100644 --- a/docs/articles_en/learn_openvino/tutorials/notebooks_section_1.rst +++ b/docs/articles_en/learn_openvino/tutorials/notebooks_section_1.rst @@ -53,11 +53,17 @@ Tutorials that explain how to optimize and quantize models with OpenVINO tools. Quantize Speech Recognition Models with accuracy control using NNCF PTQ API. +.. showcase:: + :title: 121-legacy-mo-convert-to-openvino + :img: _static/images/notebook_eye.png + + Learn about OpenVINO™ model conversion API. + .. showcase:: :title: 121-convert-to-openvino :img: _static/images/notebook_eye.png - Learn OpenVINO model conversion API. + Learn about model conversion in OpenVINO™. .. showcase:: :title: 120-tensorflow-object-detection-to-openvino diff --git a/docs/articles_en/learn_openvino/tutorials/notebooks_section_2.rst b/docs/articles_en/learn_openvino/tutorials/notebooks_section_2.rst index 522788a7311..caef030dc31 100644 --- a/docs/articles_en/learn_openvino/tutorials/notebooks_section_2.rst +++ b/docs/articles_en/learn_openvino/tutorials/notebooks_section_2.rst @@ -11,6 +11,54 @@ Model Demos Demos that demonstrate inference on a particular model. +.. showcase:: + :title: 280-depth-anything + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/280-depth-anything/280-depth-anything.gif + + Depth estimation with DepthAnything and OpenVINO. + +.. showcase:: + :title: 279-mobilevlm-language-assistant + :img: _static/images/notebook_eye.png + + Mobile language assistant with MobileVLM and OpenVINO. + +.. showcase:: + :title: 278-stable-diffusion-ip-adapter + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/278-stable-diffusion-ip-adapter/278-stable-diffusion-ip-adapter.png + + Image Generation with Stable Diffusion and IP-Adapter. + +.. showcase:: + :title: 277-amused-lightweight-text-to-image + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/277-amused-lightweight-text-to-image/277-amused-lightweight-text-to-image.png + + Lightweight image generation with aMUSEd and OpenVINO. + +.. showcase:: + :title: 276-stable-diffusion-torchdynamo-backend + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/276-stable-diffusion-torchdynamo-backend/276-stable-diffusion-torchdynamo-backend.png + + Image Generation with Stable Diffusion using OpenVINO TorchDynamo backend. + +.. showcase:: + :title: 275-llm-question-answering + :img: _static/images/notebook_eye.png + + LLM Instruction-following pipeline with OpenVINO. + +.. showcase:: + :title: 274-efficient-sam + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/274-efficient-sam/274-efficient-sam.png + + Object segmentations with EfficientSAM and OpenVINO. + +.. showcase:: + :title: 273-stable-zephyr-3b-chatbot + :img: _static/images/notebook_eye.png + + LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO. + .. showcase:: :title: 272-paint-by-example :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/272-paint-by-example/272-paint-by-example.png @@ -105,12 +153,24 @@ Demos that demonstrate inference on a particular model. Visual-language assistant with LLaVA and OpenVINO. +.. showcase:: + :title: 257-videollava-multimodal-chatbot.ipynb + :img: _static/images/notebook_eye.png + + Visual-language assistant with Video-LLaVA and OpenVINO. + .. showcase:: :title: 256-bark-text-to-audio :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/256-bark-text-to-audio/256-bark-text-to-audio.png Text-to-speech generation using Bark and OpenVINO. +.. showcase:: + :title: 254-rag-chatbot + :img: _static/images/notebook_eye.png + + Create an LLM-powered RAG system using OpenVINO. + .. showcase:: :title: 254-llm-chatbot :img: _static/images/notebook_eye.png @@ -147,9 +207,15 @@ Demos that demonstrate inference on a particular model. Universal segmentation with OneFormer and OpenVINO™. +.. showcase:: + :title: 248-segmind-vegart + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.png + + High-resolution image generation with Segmind-VegaRT and OpenVINO. + .. showcase:: :title: 248-ssd-b1 - :img: https://user-images.githubusercontent.com/29454499/258651862-28b63016-c5ff-4263-9da8-73ca31100165.jpeg + :img: https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.png Image generation with Stable Diffusion XL and OpenVINO™. diff --git a/docs/nbdoc/consts.py b/docs/nbdoc/consts.py index 3a8ca27255b..2ad6f691e5a 100644 --- a/docs/nbdoc/consts.py +++ b/docs/nbdoc/consts.py @@ -5,7 +5,7 @@ repo_directory = "notebooks" repo_owner = "openvinotoolkit" repo_name = "openvino_notebooks" repo_branch = "tree/main" -artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/" +artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/" blacklisted_extensions = ['.xml', '.bin'] notebooks_repo = "https://github.com/openvinotoolkit/openvino_notebooks/blob/main/" notebooks_binder = "https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=" diff --git a/docs/notebooks/001-hello-world-with-output.rst b/docs/notebooks/001-hello-world-with-output.rst index e52ea3af642..e40432d96ef 100644 --- a/docs/notebooks/001-hello-world-with-output.rst +++ b/docs/notebooks/001-hello-world-with-output.rst @@ -5,7 +5,7 @@ This basic introduction to OpenVINO™ shows how to do inference with an image classification model. A pre-trained `MobileNetV3 -model `__ +model `__ from `Open Model Zoo `__ is used in this tutorial. For more information about how OpenVINO IR models are @@ -13,11 +13,12 @@ created, refer to the `TensorFlow to OpenVINO <101-tensorflow-classification-to-openvino-with-output.html>`__ tutorial. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ -- `Download the Model and data samples <#download-the-model-and-data-samples>`__ +- `Download the Model and data + samples <#download-the-model-and-data-samples>`__ - `Select inference device <#select-inference-device>`__ - `Load the Model <#load-the-model>`__ - `Load an Image <#load-an-image>`__ @@ -34,8 +35,10 @@ tutorial. Note: you may need to restart the kernel to use updated packages. -Imports -------------------------------------------------- +Imports +------- + + .. code:: ipython3 @@ -55,8 +58,10 @@ Imports from notebook_utils import download_file -Download the Model and data samples ------------------------------------------------------------------------------ +Download the Model and data samples +----------------------------------- + + .. code:: ipython3 @@ -89,8 +94,10 @@ Download the Model and data samples artifacts/v3-small_224_1.0_float.bin: 0%| | 0.00/4.84M [00:00 -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/001-hello-world-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/001-hello-world-with-output_files/


../
-001-hello-world-with-output_11_1.png               31-Oct-2023 00:35              387941
-

- diff --git a/docs/notebooks/001-hello-world-with-output_files/index.html.tmp b/docs/notebooks/001-hello-world-with-output_files/index.html.tmp new file mode 100644 index 00000000000..0d4edf58fbb --- /dev/null +++ b/docs/notebooks/001-hello-world-with-output_files/index.html.tmp @@ -0,0 +1,7 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/001-hello-world-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/001-hello-world-with-output_files/


../
+001-hello-world-with-output_11_1.png               26-Jan-2024 01:05              387941
+

+ diff --git a/docs/notebooks/002-openvino-api-with-output.rst b/docs/notebooks/002-openvino-api-with-output.rst index 18bdc59b5ca..2f23557dafd 100644 --- a/docs/notebooks/002-openvino-api-with-output.rst +++ b/docs/notebooks/002-openvino-api-with-output.rst @@ -10,8 +10,8 @@ used in this tutorial are provided as examples. These model files can be replaced with your own models. The exact outputs will be different, but the process is the same. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Loading OpenVINO Runtime and Showing Info <#loading-openvino-runtime-and-showing-info>`__ @@ -57,35 +57,55 @@ the process is the same. .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. - Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0) - Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.1) - Requirement already satisfied: ipywidgets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (8.1.1) - Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.3.2) - Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.6) - Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2.1.0) - Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.11.17) - Requirement already satisfied: comm>=0.1.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (0.2.0) - Requirement already satisfied: ipython>=6.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (8.12.3) - Requirement already satisfied: traitlets>=4.3.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (5.14.0) - Requirement already satisfied: widgetsnbextension~=4.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (4.0.9) - Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (3.0.9) - Requirement already satisfied: backcall in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0) - Requirement already satisfied: decorator in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1) - Requirement already satisfied: jedi>=0.16 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1) - Requirement already satisfied: matplotlib-inline in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6) - Requirement already satisfied: pickleshare in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5) - Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.41) - Requirement already satisfied: pygments>=2.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.17.2) - Requirement already satisfied: stack-data in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.3) - Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0) - Requirement already satisfied: pexpect>4.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0) - Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3) - Requirement already satisfied: ptyprocess>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0) - Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.12) - Requirement already satisfied: executing>=1.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.0.1) - Requirement already satisfied: asttokens>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1) - Requirement already satisfied: pure-eval in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2) - Requirement already satisfied: six>=1.12.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0) + + +.. parsed-literal:: + + Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.31.0) + Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.1) + Requirement already satisfied: ipywidgets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (8.1.1) + Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.3.2) + Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (3.6) + Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2.1.0) + Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests) (2023.11.17) + + +.. parsed-literal:: + + Requirement already satisfied: comm>=0.1.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (0.2.1) + Requirement already satisfied: ipython>=6.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (8.12.3) + Requirement already satisfied: traitlets>=4.3.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (5.14.1) + Requirement already satisfied: widgetsnbextension~=4.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (4.0.9) + Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets) (3.0.9) + Requirement already satisfied: backcall in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0) + Requirement already satisfied: decorator in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1) + Requirement already satisfied: jedi>=0.16 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1) + Requirement already satisfied: matplotlib-inline in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6) + Requirement already satisfied: pickleshare in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5) + Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.43) + Requirement already satisfied: pygments>=2.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.17.2) + Requirement already satisfied: stack-data in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.3) + Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0) + + +.. parsed-literal:: + + Requirement already satisfied: pexpect>4.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0) + Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3) + Requirement already satisfied: ptyprocess>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0) + Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.13) + Requirement already satisfied: executing>=1.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.0.1) + Requirement already satisfied: asttokens>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1) + Requirement already satisfied: pure-eval in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2) + + +.. parsed-literal:: + + Requirement already satisfied: six>=1.12.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0) + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. @@ -137,7 +157,7 @@ After initializing OpenVINO Runtime, first read the model file with ``compile_model()`` method. `OpenVINO™ supports several model -formats `__ +formats `__ and enables developers to convert them to its own OpenVINO IR format using a tool dedicated to this task. @@ -198,7 +218,7 @@ notebooks. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') @@ -247,7 +267,7 @@ points to the filename of an ONNX model. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.onnx') @@ -303,7 +323,7 @@ without any conversion step. Pass the filename with extension to .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/inference.pdiparams') @@ -347,7 +367,7 @@ TensorFlow models saved in frozen graph format can also be passed to .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.pb') @@ -399,7 +419,7 @@ It is pre-trained model optimized to work with TensorFlow Lite. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.tflite') @@ -424,7 +444,7 @@ PyTorch Model `PyTorch `__ models can not be directly passed to ``core.read_model``. ``ov.Model`` for model objects from this framework can be obtained using ``ov.convert_model`` API. You can find more -details in `pytorch-to-openvino <../102-pytorch-to-openvino>`__ +details in `pytorch-to-openvino <102-pytorch-to-openvino-with-output.html>`__ notebook. In this tutorial we will use `resnet18 `__ model form torchvision library. After conversion model using @@ -473,6 +493,10 @@ Information about the inputs and outputs of the model are in .. parsed-literal:: 'model/classification.xml' already exists. + + +.. parsed-literal:: + 'model/classification.bin' already exists. @@ -480,7 +504,7 @@ Information about the inputs and outputs of the model are in .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') @@ -679,6 +703,10 @@ produced data as values. .. parsed-literal:: 'model/classification.xml' already exists. + + +.. parsed-literal:: + 'model/classification.bin' already exists. @@ -686,7 +714,7 @@ produced data as values. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') @@ -875,7 +903,7 @@ input shape. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/segmentation.bin') @@ -911,6 +939,10 @@ input shape. ~~~~ ORIGINAL MODEL ~~~~ input shape: [1,3,512,512] output shape: [1,1,512,512] + + +.. parsed-literal:: + ~~~~ RESHAPED MODEL ~~~~ model input shape: [1,3,544,544] compiled_model input shape: [1,3,544,544] @@ -1023,6 +1055,10 @@ the cache. .. parsed-literal:: 'model/classification.xml' already exists. + + +.. parsed-literal:: + 'model/classification.bin' already exists. @@ -1030,7 +1066,7 @@ the cache. .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/002-openvino-api/model/classification.bin') diff --git a/docs/notebooks/003-hello-segmentation-with-output.rst b/docs/notebooks/003-hello-segmentation-with-output.rst index 9a49528a7d0..7894b3a3f62 100644 --- a/docs/notebooks/003-hello-segmentation-with-output.rst +++ b/docs/notebooks/003-hello-segmentation-with-output.rst @@ -4,14 +4,14 @@ Hello Image Segmentation A very basic introduction to using segmentation models with OpenVINO™. In this tutorial, a pre-trained -`road-segmentation-adas-0001 `__ +`road-segmentation-adas-0001 `__ model from the `Open Model Zoo `__ is used. ADAS stands for Advanced Driver Assistance Services. The model recognizes four classes: background, road, curb and mark. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ - `Download model weights <#download-model-weights>`__ @@ -33,8 +33,10 @@ recognizes four classes: background, road, curb and mark. Note: you may need to restart the kernel to use updated packages. -Imports -------------------------------------------------- +Imports +------- + + .. code:: ipython3 @@ -42,35 +44,37 @@ Imports import matplotlib.pyplot as plt import numpy as np import openvino as ov - + # Fetch `notebook_utils` module import urllib.request urllib.request.urlretrieve( url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py', filename='notebook_utils.py' ) - + from notebook_utils import segmentation_map_to_image, download_file -Download model weights ----------------------------------------------------------------- +Download model weights +---------------------- + + .. code:: ipython3 from pathlib import Path - + base_model_dir = Path("./model").expanduser() - + model_name = "road-segmentation-adas-0001" model_xml_name = f'{model_name}.xml' model_bin_name = f'{model_name}.bin' - + model_xml_path = base_model_dir / model_xml_name - + if not model_xml_path.exists(): model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.xml" model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.bin" - + download_file(model_xml_url, model_xml_name, base_model_dir) download_file(model_bin_url, model_bin_name, base_model_dir) else: @@ -89,15 +93,17 @@ Download model weights model/road-segmentation-adas-0001.bin: 0%| | 0.00/720k [00:00`__ dataset is -provided. +A sample image from the +`Mapillary Vistas `__ dataset +is provided. .. code:: ipython3 @@ -144,23 +152,23 @@ provided. "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/empty_road_mapillary.jpg", directory="data" ) - + # The segmentation network expects images in BGR format. image = cv2.imread(str(image_filename)) - + rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) image_h, image_w, _ = image.shape - + # N,C,H,W = batch size, number of channels, height, width. N, C, H, W = input_layer_ir.shape - + # OpenCV resize expects the destination size as (width, height). resized_image = cv2.resize(image, (W, H)) - + # Reshape to the network input shape. input_image = np.expand_dims( resized_image.transpose(2, 0, 1), 0 - ) + ) plt.imshow(rgb_image) @@ -174,7 +182,7 @@ provided. .. parsed-literal:: - + @@ -182,14 +190,16 @@ provided. .. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png -Do Inference ------------------------------------------------------- +Do Inference +------------ + + .. code:: ipython3 # Run the inference. result = compiled_model([input_image])[output_layer_ir] - + # Prepare data for visualization. segmentation_mask = np.argmax(result, axis=1) plt.imshow(segmentation_mask.transpose(1, 2, 0)) @@ -199,7 +209,7 @@ Do Inference .. parsed-literal:: - + @@ -207,41 +217,45 @@ Do Inference .. image:: 003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png -Prepare Data for Visualization ------------------------------------------------------------------------- +Prepare Data for Visualization +------------------------------ + + .. code:: ipython3 # Define colormap, each color represents a class. colormap = np.array([[68, 1, 84], [48, 103, 141], [53, 183, 120], [199, 216, 52]]) - + # Define the transparency of the segmentation mask on the photo. alpha = 0.3 - + # Use function from notebook_utils.py to transform mask to an RGB image. mask = segmentation_map_to_image(segmentation_mask, colormap) resized_mask = cv2.resize(mask, (image_w, image_h)) - + # Create an image with mask. image_with_mask = cv2.addWeighted(resized_mask, alpha, rgb_image, 1 - alpha, 0) -Visualize data --------------------------------------------------------- +Visualize data +-------------- + + .. code:: ipython3 # Define titles with images. data = {"Base Photo": rgb_image, "Segmentation": mask, "Masked Photo": image_with_mask} - + # Create a subplot to visualize images. fig, axs = plt.subplots(1, len(data.items()), figsize=(15, 10)) - + # Fill the subplot. for ax, (name, image) in zip(axs, data.items()): ax.axis('off') ax.set_title(name) ax.imshow(image) - + # Display an image. plt.show(fig) diff --git a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png index 3677caabff4..b25632d5a95 100644 --- a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png +++ b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_11_2.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:76113c575caa9c8a8aca45d3ec6ebd7a4b513dadffd8e9e63861a7a041d7e5de +oid sha256:90fac1140a8f375892b9637d611b5b4d7e51f78a45f3af3ab3b33e74b97baa18 size 249032 diff --git a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png index 76c886c3c9c..bc896aa9eca 100644 --- a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png +++ b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_13_1.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9d40b840eea67857e42a78dd89b65799215e806e4f8d23b0523b0c163cb453b +oid sha256:57f846855a237159135e3491021459d4d94b97ddb0458c74b7bc35cca5ff0d5a size 20550 diff --git a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_17_0.png b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_17_0.png index 1f70e39a921..7865ccd173a 100644 --- a/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_17_0.png +++ b/docs/notebooks/003-hello-segmentation-with-output_files/003-hello-segmentation-with-output_17_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6652ac31dc4e31bcb618c0c13795dac83fa49291a436aea8058847f179b4095d +oid sha256:655bbfae2c0894ca52bb1cdaf7de64bbad747265984e968afc86f0198bc6600d size 260045 diff --git a/docs/notebooks/003-hello-segmentation-with-output_files/index.html b/docs/notebooks/003-hello-segmentation-with-output_files/index.html.tmp similarity index 51% rename from docs/notebooks/003-hello-segmentation-with-output_files/index.html rename to docs/notebooks/003-hello-segmentation-with-output_files/index.html.tmp index 150c980285e..27eec1da506 100644 --- a/docs/notebooks/003-hello-segmentation-with-output_files/index.html +++ b/docs/notebooks/003-hello-segmentation-with-output_files/index.html.tmp @@ -1,9 +1,9 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/003-hello-segmentation-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/003-hello-segmentation-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/003-hello-segmentation-with-output_files/


../
-003-hello-segmentation-with-output_11_2.png        31-Oct-2023 00:35              249032
-003-hello-segmentation-with-output_13_1.png        31-Oct-2023 00:35               20550
-003-hello-segmentation-with-output_17_0.png        31-Oct-2023 00:35              260045
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/003-hello-segmentation-with-output_files/


../
+003-hello-segmentation-with-output_11_2.png        26-Jan-2024 01:05              249032
+003-hello-segmentation-with-output_13_1.png        26-Jan-2024 01:05               20550
+003-hello-segmentation-with-output_17_0.png        26-Jan-2024 01:05              260045
 

diff --git a/docs/notebooks/004-hello-detection-with-output.rst b/docs/notebooks/004-hello-detection-with-output.rst index f1ce601889d..d81ca1ee742 100644 --- a/docs/notebooks/004-hello-detection-with-output.rst +++ b/docs/notebooks/004-hello-detection-with-output.rst @@ -5,7 +5,7 @@ A very basic introduction to using object detection models with OpenVINO™. The -`horizontal-text-detection-0001 `__ +`horizontal-text-detection-0001 `__ model from `Open Model Zoo `__ is used. It detects horizontal text in images and returns a blob of data in the @@ -16,8 +16,8 @@ corner, ``(x_max, y_max)`` are the coordinates of the bottom right bounding box corner and ``conf`` is the confidence for the predicted class. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ - `Download model weights <#download-model-weights>`__ @@ -38,8 +38,10 @@ class. Note: you may need to restart the kernel to use updated packages. -Imports -------------------------------------------------- +Imports +------- + + .. code:: ipython3 @@ -58,8 +60,10 @@ Imports from notebook_utils import download_file -Download model weights ----------------------------------------------------------------- +Download model weights +---------------------- + + .. code:: ipython3 @@ -94,8 +98,10 @@ Download model weights model/horizontal-text-detection-0001.bin: 0%| | 0.00/7.39M [00:00 -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/004-hello-detection-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/004-hello-detection-with-output_files/


../
-004-hello-detection-with-output_11_1.png           31-Oct-2023 00:35              305482
-004-hello-detection-with-output_16_0.png           31-Oct-2023 00:35              457214
-

- diff --git a/docs/notebooks/004-hello-detection-with-output_files/index.html.tmp b/docs/notebooks/004-hello-detection-with-output_files/index.html.tmp new file mode 100644 index 00000000000..cd39adc5caf --- /dev/null +++ b/docs/notebooks/004-hello-detection-with-output_files/index.html.tmp @@ -0,0 +1,8 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/004-hello-detection-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/004-hello-detection-with-output_files/


../
+004-hello-detection-with-output_11_1.png           26-Jan-2024 01:05              305482
+004-hello-detection-with-output_16_0.png           26-Jan-2024 01:05              457214
+

+ diff --git a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst index 1fb04382d1b..97e849bdb5b 100644 --- a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst +++ b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output.rst @@ -2,7 +2,7 @@ Convert a TensorFlow Model to OpenVINO™ ======================================= This short tutorial shows how to convert a TensorFlow -`MobileNetV3 `__ +`MobileNetV3 `__ image classification model to OpenVINO `Intermediate Representation `__ (OpenVINO IR) format, using `Model Conversion @@ -11,7 +11,8 @@ After creating the OpenVINO IR, load the model in `OpenVINO Runtime `__ and do inference with a sample image. -**Table of contents:** +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ - `Settings <#settings>`__ @@ -74,10 +75,14 @@ Imports .. parsed-literal:: - 2023-11-14 22:30:46.626761: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-11-14 22:30:46.661288: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + 2024-01-25 22:33:56.723840: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 22:33:56.757735: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-11-14 22:30:47.171314: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + 2024-01-25 22:33:57.273134: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Settings @@ -117,12 +122,12 @@ and save it to the disk. .. parsed-literal:: - 2023-11-14 22:30:50.201471: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW - 2023-11-14 22:30:50.201504: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07 - 2023-11-14 22:30:50.201508: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07 - 2023-11-14 22:30:50.201646: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2 - 2023-11-14 22:30:50.201662: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3 - 2023-11-14 22:30:50.201665: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration + 2024-01-25 22:34:00.136277: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW + 2024-01-25 22:34:00.136313: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07 + 2024-01-25 22:34:00.136317: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07 + 2024-01-25 22:34:00.136451: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2 + 2024-01-25 22:34:00.136466: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3 + 2024-01-25 22:34:00.136470: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration .. parsed-literal:: @@ -132,9 +137,13 @@ and save it to the disk. .. parsed-literal:: - 2023-11-14 22:30:54.370304: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024] + 2024-01-25 22:34:04.279915: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024] [[{{node inputs}}]] - 2023-11-14 22:30:57.509389: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024] + + +.. parsed-literal:: + + 2024-01-25 22:34:07.400979: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'inputs' with dtype float and shape [?,1,1,1024] [[{{node inputs}}]] WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 54). These functions will not be directly callable after loading. @@ -165,7 +174,7 @@ model directory and returns OpenVINO Model class instance which represents this model. Obtained model is ready to use and to be loaded on a device using ``ov.compile_model`` or can be saved on a disk using the ``ov.save_model`` function. See the -`tutorial `__ +`tutorial `__ for more information about using model conversion API with TensorFlow models. @@ -353,5 +362,5 @@ performance. .. parsed-literal:: - IR model in OpenVINO Runtime/CPU: 0.0010 seconds per image, FPS: 962.52 + IR model in OpenVINO Runtime/CPU: 0.0011 seconds per image, FPS: 933.95 diff --git a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_19_1.png b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_19_1.png index 15c5ba574c1..3f302494abc 100644 --- a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_19_1.png +++ b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/101-tensorflow-classification-to-openvino-with-output_19_1.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7511b8a4e5b047600d5fed14fbc7e9653a868bc5253abf1e0c3ef649b47bc408 +oid sha256:2dd4338c6c163e7693885ce544e8c9cd2aecedf3b136fa295e22877f37b5634c size 387941 diff --git a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html.tmp similarity index 58% rename from docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html rename to docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html.tmp index 50f5758c278..fec1f1be4eb 100644 --- a/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html +++ b/docs/notebooks/101-tensorflow-classification-to-openvino-with-output_files/index.html.tmp @@ -1,7 +1,7 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/


../
-101-tensorflow-classification-to-openvino-with-..> 15-Nov-2023 00:43              387941
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/101-tensorflow-classification-to-openvino-with-output_files/


../
+101-tensorflow-classification-to-openvino-with-..> 26-Jan-2024 01:05              387941
 

diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst index 0155fb02753..96fa90f1545 100644 --- a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst +++ b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output.rst @@ -33,8 +33,8 @@ plant, sheep, sofa, train, tv monitor** More information about the model is available in the `torchvision documentation `__ -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Preparation <#preparation>`__ @@ -56,10 +56,10 @@ documentation `__ model <#load-the-openvino-ir-network-and-run-inference-on-the-onnx-model>`__ - `1. ONNX Model in OpenVINO - Runtime <#-onnx-model-in-openvino-runtime>`__ + Runtime <#1--onnx-model-in-openvino-runtime>`__ - `Select inference device <#select-inference-device>`__ - `2. OpenVINO IR Model in OpenVINO - Runtime <#-openvino-ir-model-in-openvino-runtime>`__ + Runtime <#2--openvino-ir-model-in-openvino-runtime>`__ - `Select inference device <#select-inference-device>`__ - `PyTorch Comparison <#pytorch-comparison>`__ @@ -576,9 +576,17 @@ performance. .. parsed-literal:: - PyTorch model on CPU: 0.039 seconds per image, FPS: 25.93 - ONNX model in OpenVINO Runtime/CPU: 0.018 seconds per image, FPS: 56.39 - OpenVINO IR model in OpenVINO Runtime/CPU: 0.018 seconds per image, FPS: 54.58 + PyTorch model on CPU: 0.042 seconds per image, FPS: 24.02 + + +.. parsed-literal:: + + ONNX model in OpenVINO Runtime/CPU: 0.030 seconds per image, FPS: 33.66 + + +.. parsed-literal:: + + OpenVINO IR model in OpenVINO Runtime/CPU: 0.029 seconds per image, FPS: 35.01 **Show Device Information** @@ -610,4 +618,4 @@ References - `Model Conversion API documentation `__ - `Converting Pytorch - model `__ + model `__ diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_22_0.png b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_22_0.png index 945e25fb7ae..7a126b5a035 100644 --- a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_22_0.png +++ b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_22_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c48d5afdee0d49838332deac5fea09e48ea455371a5bc88fd865b070fcffbc3 +oid sha256:bdc636d912f3e7146a2aacdaf744a83a90ad66778eacac5924a2499d8acb351f size 465692 diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_27_0.png b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_27_0.png index 15e166b09bb..96d76cdb270 100644 --- a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_27_0.png +++ b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_27_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb50dbc3d8a370e69c72cedea217cb413f04852f64580e0c7b54aa495b6c5ad2 +oid sha256:1481c5d43590bbae0644ab25ea265919fb1d8ecd7e4f628e25e2ef69e822c4f0 size 465695 diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_29_0.png b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_29_0.png index 945e25fb7ae..7a126b5a035 100644 --- a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_29_0.png +++ b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/102-pytorch-onnx-to-openvino-with-output_29_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c48d5afdee0d49838332deac5fea09e48ea455371a5bc88fd865b070fcffbc3 +oid sha256:bdc636d912f3e7146a2aacdaf744a83a90ad66778eacac5924a2499d8acb351f size 465692 diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html deleted file mode 100644 index 0030b3e836c..00000000000 --- a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html +++ /dev/null @@ -1,9 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/


../
-102-pytorch-onnx-to-openvino-with-output_22_0.png  31-Oct-2023 00:35              465692
-102-pytorch-onnx-to-openvino-with-output_27_0.png  31-Oct-2023 00:35              465695
-102-pytorch-onnx-to-openvino-with-output_29_0.png  31-Oct-2023 00:35              465692
-

- diff --git a/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html.tmp b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html.tmp new file mode 100644 index 00000000000..9ac41953398 --- /dev/null +++ b/docs/notebooks/102-pytorch-onnx-to-openvino-with-output_files/index.html.tmp @@ -0,0 +1,9 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-onnx-to-openvino-with-output_files/


../
+102-pytorch-onnx-to-openvino-with-output_22_0.png  26-Jan-2024 01:05              465692
+102-pytorch-onnx-to-openvino-with-output_27_0.png  26-Jan-2024 01:05              465695
+102-pytorch-onnx-to-openvino-with-output_29_0.png  26-Jan-2024 01:05              465692
+

+ diff --git a/docs/notebooks/102-pytorch-to-openvino-with-output.rst b/docs/notebooks/102-pytorch-to-openvino-with-output.rst new file mode 100644 index 00000000000..25a97931195 --- /dev/null +++ b/docs/notebooks/102-pytorch-to-openvino-with-output.rst @@ -0,0 +1,830 @@ +Convert a PyTorch Model to OpenVINO™ IR +======================================= + +This tutorial demonstrates step-by-step instructions on how to do +inference on a PyTorch classification model using OpenVINO Runtime. +Starting from OpenVINO 2023.0 release, OpenVINO supports direct PyTorch +model conversion without an intermediate step to convert them into ONNX +format. In order, if you try to use the lower OpenVINO version or prefer +to use ONNX, please check this +`tutorial <102-pytorch-to-openvino-with-output.html>`__. + +In this tutorial, we will use the +`RegNetY_800MF `__ model from +`torchvision `__ to +demonstrate how to convert PyTorch models to OpenVINO Intermediate +Representation. + +The RegNet model was proposed in `Designing Network Design +Spaces `__ by Ilija Radosavovic, Raj +Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár. The authors +design search spaces to perform Neural Architecture Search (NAS). They +first start from a high dimensional search space and iteratively reduce +the search space by empirically applying constraints based on the +best-performing models sampled by the current search space. Instead of +focusing on designing individual network instances, authors design +network design spaces that parametrize populations of networks. The +overall process is analogous to the classic manual design of networks +but elevated to the design space level. The RegNet design space provides +simple and fast networks that work well across a wide range of flop +regimes. + +Table of contents: +^^^^^^^^^^^^^^^^^^ + +- `Prerequisites <#prerequisites>`__ +- `Load PyTorch Model <#load-pytorch-model>`__ + + - `Prepare Input Data <#prepare-input-data>`__ + - `Run PyTorch Model Inference <#run-pytorch-model-inference>`__ + - `Benchmark PyTorch Model + Inference <#benchmark-pytorch-model-inference>`__ + +- `Convert PyTorch Model to OpenVINO Intermediate + Representation <#convert-pytorch-model-to-openvino-intermediate-representation>`__ + + - `Select inference device <#select-inference-device>`__ + - `Run OpenVINO Model Inference <#run-openvino-model-inference>`__ + - `Benchmark OpenVINO Model + Inference <#benchmark-openvino-model-inference>`__ + +- `Convert PyTorch Model with Static Input + Shape <#convert-pytorch-model-with-static-input-shape>`__ + + - `Select inference device <#select-inference-device>`__ + - `Run OpenVINO Model Inference with Static Input + Shape <#run-openvino-model-inference-with-static-input-shape>`__ + - `Benchmark OpenVINO Model Inference with Static Input + Shape <#benchmark-openvino-model-inference-with-static-input-shape>`__ + +- `Convert TorchScript Model to OpenVINO Intermediate + Representation <#convert-torchscript-model-to-openvino-intermediate-representation>`__ + + - `Scripted Model <#scripted-model>`__ + - `Benchmark Scripted Model + Inference <#benchmark-scripted-model-inference>`__ + - `Convert PyTorch Scripted Model to OpenVINO Intermediate + Representation <#convert-pytorch-scripted-model-to-openvino-intermediate-representation>`__ + - `Benchmark OpenVINO Model Inference Converted From Scripted + Model <#benchmark-openvino-model-inference-converted-from-scripted-model>`__ + - `Traced Model <#traced-model>`__ + - `Benchmark Traced Model + Inference <#benchmark-traced-model-inference>`__ + - `Convert PyTorch Traced Model to OpenVINO Intermediate + Representation <#convert-pytorch-traced-model-to-openvino-intermediate-representation>`__ + - `Benchmark OpenVINO Model Inference Converted From Traced + Model <#benchmark-openvino-model-inference-converted-from-traced-model>`__ + +Prerequisites +------------- + + + +Install notebook dependencies + +.. code:: ipython3 + + %pip install -q "openvino>=2023.1.0" scipy + + +.. parsed-literal:: + + Note: you may need to restart the kernel to use updated packages. + + +Download input data and label map + +.. code:: ipython3 + + import requests + from pathlib import Path + from PIL import Image + + MODEL_DIR = Path("model") + DATA_DIR = Path("data") + + MODEL_DIR.mkdir(exist_ok=True) + DATA_DIR.mkdir(exist_ok=True) + MODEL_NAME = "regnet_y_800mf" + + image = Image.open(requests.get("https://farm9.staticflickr.com/8225/8511402100_fea15da1c5_z.jpg", stream=True).raw) + + labels_file = DATA_DIR / "imagenet_2012.txt" + + if not labels_file.exists(): + resp = requests.get("https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/imagenet_2012.txt") + with labels_file.open("wb") as f: + f.write(resp.content) + + imagenet_classes = labels_file.open("r").read().splitlines() + +Load PyTorch Model +------------------ + + + +Generally, PyTorch models represent an instance of the +``torch.nn.Module`` class, initialized by a state dictionary with model +weights. Typical steps for getting a pre-trained model: + +1. Create an instance of a model class +2. Load checkpoint state dict, which contains pre-trained model weights +3. Turn the model to evaluation for switching some operations to + inference mode + +The ``torchvision`` module provides a ready-to-use set of functions for +model class initialization. We will use +``torchvision.models.regnet_y_800mf``. You can directly pass pre-trained +model weights to the model initialization function using the weights +enum ``RegNet_Y_800MF_Weights.DEFAULT``. + +.. code:: ipython3 + + import torchvision + + # get default weights using available weights Enum for model + weights = torchvision.models.RegNet_Y_800MF_Weights.DEFAULT + + # create model topology and load weights + model = torchvision.models.regnet_y_800mf(weights=weights) + + # switch model to inference mode + model.eval(); + +Prepare Input Data +~~~~~~~~~~~~~~~~~~ + + + +The code below demonstrates how to preprocess input data using a +model-specific transforms module from ``torchvision``. After +transformation, we should concatenate images into batched tensor, in our +case, we will run the model with batch 1, so we just unsqueeze input on +the first dimension. + +.. code:: ipython3 + + import torch + + # Initialize the Weight Transforms + preprocess = weights.transforms() + + # Apply it to the input image + img_transformed = preprocess(image) + + # Add batch dimension to image tensor + input_tensor = img_transformed.unsqueeze(0) + +Run PyTorch Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +The model returns a vector of probabilities in raw logits format, +softmax can be applied to get normalized values in the [0, 1] range. For +a demonstration that the output of the original model and OpenVINO +converted is the same, we defined a common postprocessing function which +can be reused later. + +.. code:: ipython3 + + import numpy as np + from scipy.special import softmax + + # Perform model inference on input tensor + result = model(input_tensor) + + # Postprocessing function for getting results in the same way for both PyTorch model inference and OpenVINO + def postprocess_result(output_tensor:np.ndarray, top_k:int = 5): + """ + Posprocess model results. This function applied sofrmax on output tensor and returns specified top_k number of labels with highest probability + Parameters: + output_tensor (np.ndarray): model output tensor with probabilities + top_k (int, *optional*, default 5): number of labels with highest probability for return + Returns: + topk_labels: label ids for selected top_k scores + topk_scores: selected top_k highest scores predicted by model + """ + softmaxed_scores = softmax(output_tensor, -1)[0] + topk_labels = np.argsort(softmaxed_scores)[-top_k:][::-1] + topk_scores = softmaxed_scores[topk_labels] + return topk_labels, topk_scores + + # Postprocess results + top_labels, top_scores = postprocess_result(result.detach().numpy()) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_11_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark PyTorch Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + # Run model inference + model(input_tensor) + + +.. parsed-literal:: + + 16.4 ms ± 673 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Convert PyTorch Model to OpenVINO Intermediate Representation +------------------------------------------------------------- + + + +Starting from the 2023.0 release OpenVINO supports direct PyTorch models +conversion to OpenVINO Intermediate Representation (IR) format. OpenVINO +model conversion API should be used for these purposes. More details +regarding PyTorch model conversion can be found in OpenVINO +`documentation `__ + +The ``convert_model`` function accepts the PyTorch model object and +returns the ``openvino.Model`` instance ready to load on a device using +``core.compile_model`` or save on disk for next usage using +``ov.save_model``. Optionally, we can provide additional parameters, +such as: + +- ``compress_to_fp16`` - flag to perform model weights compression into + FP16 data format. It may reduce the required space for model storage + on disk and give speedup for inference devices, where FP16 + calculation is supported. +- ``example_input`` - input data sample which can be used for model + tracing. +- ``input_shape`` - the shape of input tensor for conversion + +and any other advanced options supported by model conversion Python API. +More details can be found on this +`page `__ + +.. code:: ipython3 + + import openvino as ov + + # Create OpenVINO Core object instance + core = ov.Core() + + # Convert model to openvino.runtime.Model object + ov_model = ov.convert_model(model) + + # Save openvino.runtime.Model object on disk + ov.save_model(ov_model, MODEL_DIR / f"{MODEL_NAME}_dynamic.xml") + + ov_model + + + + +.. parsed-literal:: + + + ] + outputs[ + + ]> + + + +Select inference device +~~~~~~~~~~~~~~~~~~~~~~~ + + + +select device from dropdown list for running inference using OpenVINO + +.. code:: ipython3 + + import ipywidgets as widgets + + device = widgets.Dropdown( + options=core.available_devices + ["AUTO"], + value='AUTO', + description='Device:', + disabled=False, + ) + + device + + + + +.. parsed-literal:: + + Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') + + + +.. code:: ipython3 + + # Load OpenVINO model on device + compiled_model = core.compile_model(ov_model, device.value) + compiled_model + + + + +.. parsed-literal:: + + + ] + outputs[ + + ]> + + + +Run OpenVINO Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + # Run model inference + result = compiled_model(input_tensor)[0] + + # Posptorcess results + top_labels, top_scores = postprocess_result(result) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_20_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark OpenVINO Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + compiled_model(input_tensor) + + +.. parsed-literal:: + + 3.31 ms ± 28.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Convert PyTorch Model with Static Input Shape +--------------------------------------------- + + + +The default conversion path preserves dynamic input shapes, in order if +you want to convert the model with static shapes, you can explicitly +specify it during conversion using the ``input_shape`` parameter or +reshape the model into the desired shape after conversion. For the model +reshaping example please check the following +`tutorial <002-openvino-api-with-output.html>`__. + +.. code:: ipython3 + + # Convert model to openvino.runtime.Model object + ov_model = ov.convert_model(model, input=[[1,3,224,224]]) + # Save openvino.runtime.Model object on disk + ov.save_model(ov_model, MODEL_DIR / f"{MODEL_NAME}_static.xml") + ov_model + + + + +.. parsed-literal:: + + + ] + outputs[ + + ]> + + + +Select inference device +~~~~~~~~~~~~~~~~~~~~~~~ + + + +select device from dropdown list for running inference using OpenVINO + +.. code:: ipython3 + + device + + + + +.. parsed-literal:: + + Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') + + + +.. code:: ipython3 + + # Load OpenVINO model on device + compiled_model = core.compile_model(ov_model, device.value) + compiled_model + + + + +.. parsed-literal:: + + + ] + outputs[ + + ]> + + + +Now, we can see that input of our converted model is tensor of shape [1, +3, 224, 224] instead of [?, 3, ?, ?] reported by previously converted +model. + +Run OpenVINO Model Inference with Static Input Shape +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + # Run model inference + result = compiled_model(input_tensor)[0] + + # Posptorcess results + top_labels, top_scores = postprocess_result(result) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_31_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark OpenVINO Model Inference with Static Input Shape +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + compiled_model(input_tensor) + + +.. parsed-literal:: + + 2.89 ms ± 38.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Convert TorchScript Model to OpenVINO Intermediate Representation +----------------------------------------------------------------- + + + +TorchScript is a way to create serializable and optimizable models from +PyTorch code. Any TorchScript program can be saved from a Python process +and loaded in a process where there is no Python dependency. More +details about TorchScript can be found in `PyTorch +documentation `__. + +There are 2 possible ways to convert the PyTorch model to TorchScript: + +- ``torch.jit.script`` - Scripting a function or ``nn.Module`` will + inspect the source code, compile it as TorchScript code using the + TorchScript compiler, and return a ``ScriptModule`` or + ``ScriptFunction``. +- ``torch.jit.trace`` - Trace a function and return an executable or + ``ScriptFunction`` that will be optimized using just-in-time + compilation. + +Let’s consider both approaches and their conversion into OpenVINO IR. + +Scripted Model +~~~~~~~~~~~~~~ + + + +``torch.jit.script`` inspects model source code and compiles it to +``ScriptModule``. After compilation model can be used for inference or +saved on disk using the ``torch.jit.save`` function and after that +restored with ``torch.jit.load`` in any other environment without the +original PyTorch model code definitions. + +TorchScript itself is a subset of the Python language, so not all +features in Python work, but TorchScript provides enough functionality +to compute on tensors and do control-dependent operations. For a +complete guide, see the `TorchScript Language +Reference `__. + +.. code:: ipython3 + + # Get model path + scripted_model_path = MODEL_DIR / f"{MODEL_NAME}_scripted.pth" + + # Compile and save model if it has not been compiled before or load compiled model + if not scripted_model_path.exists(): + scripted_model = torch.jit.script(model) + torch.jit.save(scripted_model, scripted_model_path) + else: + scripted_model = torch.jit.load(scripted_model_path) + + # Run scripted model inference + result = scripted_model(input_tensor) + + # Postprocess results + top_labels, top_scores = postprocess_result(result.detach().numpy()) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_35_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark Scripted Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + scripted_model(input_tensor) + + +.. parsed-literal:: + + 12.8 ms ± 6.97 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Convert PyTorch Scripted Model to OpenVINO Intermediate Representation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +The conversion step for the scripted model to OpenVINO IR is similar to +the original PyTorch model. + +.. code:: ipython3 + + # Convert model to openvino.runtime.Model object + ov_model = ov.convert_model(scripted_model) + + # Load OpenVINO model on device + compiled_model = core.compile_model(ov_model, device.value) + + # Run OpenVINO model inference + result = compiled_model(input_tensor, device.value)[0] + + # Postprocess results + top_labels, top_scores = postprocess_result(result) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_39_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark OpenVINO Model Inference Converted From Scripted Model +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + compiled_model(input_tensor) + + +.. parsed-literal:: + + 3.41 ms ± 6.84 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Traced Model +~~~~~~~~~~~~ + + + +Using ``torch.jit.trace``, you can turn an existing module or Python +function into a TorchScript ``ScriptFunction`` or ``ScriptModule``. You +must provide example inputs, and model will be executed, recording the +operations performed on all the tensors. + +- The resulting recording of a standalone function produces + ``ScriptFunction``. + +- The resulting recording of ``nn.Module.forward`` or ``nn.Module`` + produces ``ScriptModule``. + +In the same way like scripted model, traced model can be used for +inference or saved on disk using ``torch.jit.save`` function and after +that restored with ``torch.jit.load`` in any other environment without +original PyTorch model code definitions. + +.. code:: ipython3 + + # Get model path + traced_model_path = MODEL_DIR / f"{MODEL_NAME}_traced.pth" + + # Trace and save model if it has not been traced before or load traced model + if not traced_model_path.exists(): + traced_model = torch.jit.trace(model, example_inputs=input_tensor) + torch.jit.save(traced_model, traced_model_path) + else: + traced_model = torch.jit.load(traced_model_path) + + # Run traced model inference + result = traced_model(input_tensor) + + # Postprocess results + top_labels, top_scores = postprocess_result(result.detach().numpy()) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_43_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark Traced Model Inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + traced_model(input_tensor) + + +.. parsed-literal:: + + 12.2 ms ± 29 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + + +Convert PyTorch Traced Model to OpenVINO Intermediate Representation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +The conversion step for a traced model to OpenVINO IR is similar to the +original PyTorch model. + +.. code:: ipython3 + + # Convert model to openvino.runtime.Model object + ov_model = ov.convert_model(traced_model) + + # Load OpenVINO model on device + compiled_model = core.compile_model(ov_model, device.value) + + # Run OpenVINO model inference + result = compiled_model(input_tensor)[0] + + # Postprocess results + top_labels, top_scores = postprocess_result(result) + + # Show results + display(image) + for idx, (label, score) in enumerate(zip(top_labels, top_scores)): + _, predicted_label = imagenet_classes[label].split(" ", 1) + print(f"{idx + 1}: {predicted_label} - {score * 100 :.2f}%") + + + +.. image:: 102-pytorch-to-openvino-with-output_files/102-pytorch-to-openvino-with-output_47_0.png + + +.. parsed-literal:: + + 1: tiger cat - 25.91% + 2: Egyptian cat - 10.26% + 3: computer keyboard, keypad - 9.22% + 4: tabby, tabby cat - 9.09% + 5: hamper - 2.35% + + +Benchmark OpenVINO Model Inference Converted From Traced Model +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +.. code:: ipython3 + + %%timeit + + compiled_model(input_tensor)[0] + + +.. parsed-literal:: + + 3.4 ms ± 3.43 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) + diff --git a/docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html b/docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html.tmp similarity index 55% rename from docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html rename to docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html.tmp index 953c8eb9d09..5e046f8bcd6 100644 --- a/docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html +++ b/docs/notebooks/102-pytorch-to-openvino-with-output_files/index.html.tmp @@ -1,20 +1,20 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/102-pytorch-to-openvino-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-to-openvino-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/102-pytorch-to-openvino-with-output_files/


../
-102-pytorch-to-openvino-with-output_11_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_11_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_20_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_20_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_31_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_31_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_35_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_35_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_39_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_39_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_43_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_43_0.png       07-Dec-2023 00:49              542516
-102-pytorch-to-openvino-with-output_47_0.jpg       07-Dec-2023 00:49               54874
-102-pytorch-to-openvino-with-output_47_0.png       07-Dec-2023 00:49              542516
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/102-pytorch-to-openvino-with-output_files/


../
+102-pytorch-to-openvino-with-output_11_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_11_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_20_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_20_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_31_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_31_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_35_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_35_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_39_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_39_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_43_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_43_0.png       26-Jan-2024 01:05              542516
+102-pytorch-to-openvino-with-output_47_0.jpg       26-Jan-2024 01:05               54874
+102-pytorch-to-openvino-with-output_47_0.png       26-Jan-2024 01:05              542516
 

diff --git a/docs/notebooks/103-paddle-to-openvino-classification-with-output.rst b/docs/notebooks/103-paddle-to-openvino-classification-with-output.rst index 4b22013af12..7934c400591 100644 --- a/docs/notebooks/103-paddle-to-openvino-classification-with-output.rst +++ b/docs/notebooks/103-paddle-to-openvino-classification-with-output.rst @@ -14,8 +14,8 @@ IR model. Source of the `model `__. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Preparation <#preparation>`__ @@ -45,28 +45,48 @@ Imports .. code:: ipython3 - %pip install -q "paddlepaddle>=2.5.1" + import platform + + if platform.system() == "Windows": + %pip install -q "paddlepaddle>=2.5.1,<2.6.0" + else: + %pip install -q "paddlepaddle>=2.5.1" %pip install -q paddleclas --no-deps %pip install -q "prettytable" "ujson" "visualdl>=2.2.0" "faiss-cpu>=1.7.1" # Install openvino package - !pip install -q "openvino>=2023.1.0" + %pip install -q "openvino>=2023.1.0" .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. paddleclas 2.5.1 requires easydict, which is not installed. paddleclas 2.5.1 requires faiss-cpu==1.7.1.post2, but you have faiss-cpu 1.7.4 which is incompatible. paddleclas 2.5.1 requires gast==0.3.3, but you have gast 0.4.0 which is incompatible. + + +.. parsed-literal:: + + Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. .. code:: ipython3 - import platform - if (platform.system() == "Linux"): !wget http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb !sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb @@ -74,12 +94,20 @@ Imports .. parsed-literal:: - --2023-12-06 22:32:58-- http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb + --2024-01-25 22:35:57-- http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb Resolving proxy-mu.intel.com (proxy-mu.intel.com)... 10.217.247.236 Connecting to proxy-mu.intel.com (proxy-mu.intel.com)|10.217.247.236|:911... connected. - Proxy request sent, awaiting response... 404 Not Found - 2023-12-06 22:32:59 ERROR 404: Not Found. + Proxy request sent, awaiting response... + +.. parsed-literal:: + + 404 Not Found + 2024-01-25 22:35:57 ERROR 404: Not Found. + + +.. parsed-literal:: + dpkg: error: cannot access archive 'libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb': No such file or directory @@ -107,8 +135,12 @@ Imports .. parsed-literal:: - 2023-12-06 22:33:00 INFO: Loading faiss with AVX2 support. - 2023-12-06 22:33:00 INFO: Successfully loaded faiss with AVX2 support. + 2024-01-25 22:35:59 INFO: Loading faiss with AVX2 support. + + +.. parsed-literal:: + + 2024-01-25 22:35:59 INFO: Successfully loaded faiss with AVX2 support. Settings @@ -192,7 +224,11 @@ inference on that image, and then show the top three prediction results. .. parsed-literal:: - [2023/12/06 22:33:21] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead. + [2024/01/25 22:36:20] ppcls WARNING: The current running environment does not support the use of GPU. CPU has been used instead. + + +.. parsed-literal:: + Labrador retriever, 0.75138 German short-haired pointer, 0.02373 Great Dane, 0.01848 @@ -201,7 +237,7 @@ inference on that image, and then show the top three prediction results. -.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_1.png +.. image:: 103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_2.png ``classifier.predict()`` takes an image file name, reads the image, @@ -258,7 +294,7 @@ clipping values. .. parsed-literal:: - 2023-12-06 22:33:22 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). + 2024-01-25 22:36:20 WARNING: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). .. parsed-literal:: @@ -270,7 +306,7 @@ clipping values. .. parsed-literal:: - + @@ -453,7 +489,7 @@ Note that many optimizations are possible to improve the performance. .. parsed-literal:: - PaddlePaddle model on CPU: 0.0070 seconds per image, FPS: 142.41 + PaddlePaddle model on CPU: 0.0073 seconds per image, FPS: 137.36 PaddlePaddle result: Labrador retriever, 0.75138 @@ -517,7 +553,7 @@ select device from dropdown list for running inference using OpenVINO .. parsed-literal:: - OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0028 seconds per image, FPS: 352.29 + OpenVINO IR model in OpenVINO Runtime (AUTO): 0.0031 seconds per image, FPS: 322.19 OpenVINO result: Labrador retriever, 0.74909 @@ -538,4 +574,4 @@ References - `PaddleClas `__ - `OpenVINO PaddlePaddle - support `__ + support `__ diff --git a/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_1.png b/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_2.png similarity index 100% rename from docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_1.png rename to docs/notebooks/103-paddle-to-openvino-classification-with-output_files/103-paddle-to-openvino-classification-with-output_8_2.png diff --git a/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html b/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html deleted file mode 100644 index 66c78f6a2a4..00000000000 --- a/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html +++ /dev/null @@ -1,11 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/


../
-103-paddle-to-openvino-classification-with-outp..> 07-Dec-2023 00:49              120883
-103-paddle-to-openvino-classification-with-outp..> 07-Dec-2023 00:49              224886
-103-paddle-to-openvino-classification-with-outp..> 07-Dec-2023 00:49              224886
-103-paddle-to-openvino-classification-with-outp..> 07-Dec-2023 00:49              224886
-103-paddle-to-openvino-classification-with-outp..> 07-Dec-2023 00:49              224886
-

- diff --git a/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html.tmp b/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html.tmp new file mode 100644 index 00000000000..6165476573d --- /dev/null +++ b/docs/notebooks/103-paddle-to-openvino-classification-with-output_files/index.html.tmp @@ -0,0 +1,11 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/103-paddle-to-openvino-classification-with-output_files/


../
+103-paddle-to-openvino-classification-with-outp..> 26-Jan-2024 01:05              120883
+103-paddle-to-openvino-classification-with-outp..> 26-Jan-2024 01:05              224886
+103-paddle-to-openvino-classification-with-outp..> 26-Jan-2024 01:05              224886
+103-paddle-to-openvino-classification-with-outp..> 26-Jan-2024 01:05              224886
+103-paddle-to-openvino-classification-with-outp..> 26-Jan-2024 01:05              224886
+

+ diff --git a/docs/notebooks/104-model-tools-with-output.rst b/docs/notebooks/104-model-tools-with-output.rst index 29ca485e8e9..e3befdb2570 100644 --- a/docs/notebooks/104-model-tools-with-output.rst +++ b/docs/notebooks/104-model-tools-with-output.rst @@ -6,8 +6,8 @@ Zoo `__, convert it to OpenVINO™ IR format, show information about the model, and benchmark the model. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `OpenVINO and Open Model Zoo Tools <#openvino-and-open-model-zoo-tools>`__ @@ -15,8 +15,7 @@ the model. - `Model Name <#model-name>`__ - `Imports <#imports>`__ - - `Settings and - Configuration <#settings-and-configuration>`__ + - `Settings and Configuration <#settings-and-configuration>`__ - `Download a Model from Open Model Zoo <#download-a-model-from-open-model-zoo>`__ @@ -28,8 +27,10 @@ the model. - `Benchmark with Different Settings <#benchmark-with-different-settings>`__ -OpenVINO and Open Model Zoo Tools ---------------------------------------------------------------------------- +OpenVINO and Open Model Zoo Tools +--------------------------------- + + OpenVINO and Open Model Zoo tools are listed in the table below. @@ -60,11 +61,15 @@ OpenVINO and Open Model Zoo tools are listed in the table below. Note: you may need to restart the kernel to use updated packages. -Preparation ------------------------------------------------------ +Preparation +----------- + + + +Model Name +~~~~~~~~~~ + -Model Name -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Set ``model_name`` to the name of the Open Model Zoo model to use in this notebook. Refer to the list of @@ -79,17 +84,19 @@ pre-trained models for a full list of models that can be used. Set # model_name = "resnet-50-pytorch" model_name = "mobilenet-v2-pytorch" -Imports -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Imports +~~~~~~~ + + .. code:: ipython3 import json from pathlib import Path - + import openvino as ov from IPython.display import Markdown, display - + # Fetch `notebook_utils` module import urllib.request urllib.request.urlretrieve( @@ -98,8 +105,10 @@ Imports ) from notebook_utils import DeviceNotFoundAlert, NotebookAlert -Settings and Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Settings and Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Set the file and directory paths. By default, this notebook downloads models from Open Model Zoo to the ``open_model_zoo_models`` directory in @@ -122,11 +131,11 @@ The following settings can be changed: base_model_dir = Path("model") omz_cache_dir = Path("cache") precision = "FP16" - + # Check if an iGPU is available on this system to use with Benchmark App. core = ov.Core() gpu_available = "GPU" in core.available_devices - + print( f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}" ) @@ -137,8 +146,10 @@ The following settings can be changed: base_model_dir: model, omz_cache_dir: cache, gpu_availble: False -Download a Model from Open Model Zoo ------------------------------------------------------------------------------- +Download a Model from Open Model Zoo +------------------------------------ + + Specify, display and run the Model Downloader command to download the model. @@ -146,7 +157,7 @@ model. .. code:: ipython3 ## Uncomment the next line to show help in omz_downloader which explains the command-line options. - + # !omz_downloader --help .. code:: ipython3 @@ -171,14 +182,664 @@ Downloading mobilenet-v2-pytorch… .. parsed-literal:: ################|| Downloading mobilenet-v2-pytorch ||################ - + ========== Downloading model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth - - -Convert a Model to OpenVINO IR format -------------------------------------------------------------------------------- +.. parsed-literal:: + + ... 0%, 32 KB, 1358 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 0%, 64 KB, 1162 KB/s, 0 seconds passed +... 0%, 96 KB, 1715 KB/s, 0 seconds passed +... 0%, 128 KB, 1454 KB/s, 0 seconds passed +... 1%, 160 KB, 1683 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 1%, 192 KB, 1604 KB/s, 0 seconds passed +... 1%, 224 KB, 1859 KB/s, 0 seconds passed +... 1%, 256 KB, 2119 KB/s, 0 seconds passed +... 2%, 288 KB, 2263 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 2%, 320 KB, 2104 KB/s, 0 seconds passed +... 2%, 352 KB, 2304 KB/s, 0 seconds passed +... 2%, 384 KB, 2507 KB/s, 0 seconds passed +... 2%, 416 KB, 2607 KB/s, 0 seconds passed +... 3%, 448 KB, 2428 KB/s, 0 seconds passed +... 3%, 480 KB, 2591 KB/s, 0 seconds passed +... 3%, 512 KB, 2757 KB/s, 0 seconds passed +... 3%, 544 KB, 2833 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 4%, 576 KB, 2656 KB/s, 0 seconds passed +... 4%, 608 KB, 2792 KB/s, 0 seconds passed +... 4%, 640 KB, 2932 KB/s, 0 seconds passed +... 4%, 672 KB, 2994 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 5%, 704 KB, 2822 KB/s, 0 seconds passed +... 5%, 736 KB, 2941 KB/s, 0 seconds passed +... 5%, 768 KB, 3062 KB/s, 0 seconds passed +... 5%, 800 KB, 3113 KB/s, 0 seconds passed +... 5%, 832 KB, 2953 KB/s, 0 seconds passed +... 6%, 864 KB, 3057 KB/s, 0 seconds passed +... 6%, 896 KB, 3163 KB/s, 0 seconds passed +... 6%, 928 KB, 3207 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 6%, 960 KB, 3055 KB/s, 0 seconds passed +... 7%, 992 KB, 3149 KB/s, 0 seconds passed +... 7%, 1024 KB, 3243 KB/s, 0 seconds passed +... 7%, 1056 KB, 3283 KB/s, 0 seconds passed +... 7%, 1088 KB, 3140 KB/s, 0 seconds passed +... 8%, 1120 KB, 3224 KB/s, 0 seconds passed +... 8%, 1152 KB, 3309 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 8%, 1184 KB, 3342 KB/s, 0 seconds passed +... 8%, 1216 KB, 3208 KB/s, 0 seconds passed +... 8%, 1248 KB, 3284 KB/s, 0 seconds passed +... 9%, 1280 KB, 3362 KB/s, 0 seconds passed +... 9%, 1312 KB, 3393 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 9%, 1344 KB, 3266 KB/s, 0 seconds passed +... 9%, 1376 KB, 3336 KB/s, 0 seconds passed +... 10%, 1408 KB, 3407 KB/s, 0 seconds passed +... 10%, 1440 KB, 3304 KB/s, 0 seconds passed +... 10%, 1472 KB, 3315 KB/s, 0 seconds passed +... 10%, 1504 KB, 3380 KB/s, 0 seconds passed +... 11%, 1536 KB, 3446 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 11%, 1568 KB, 3349 KB/s, 0 seconds passed +... 11%, 1600 KB, 3358 KB/s, 0 seconds passed +... 11%, 1632 KB, 3418 KB/s, 0 seconds passed +... 11%, 1664 KB, 3478 KB/s, 0 seconds passed +... 12%, 1696 KB, 3506 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 12%, 1728 KB, 3398 KB/s, 0 seconds passed +... 12%, 1760 KB, 3454 KB/s, 0 seconds passed +... 12%, 1792 KB, 3509 KB/s, 0 seconds passed +... 13%, 1824 KB, 3529 KB/s, 0 seconds passed +... 13%, 1856 KB, 3431 KB/s, 0 seconds passed +... 13%, 1888 KB, 3484 KB/s, 0 seconds passed +... 13%, 1920 KB, 3534 KB/s, 0 seconds passed +... 14%, 1952 KB, 3555 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 14%, 1984 KB, 3458 KB/s, 0 seconds passed +... 14%, 2016 KB, 3510 KB/s, 0 seconds passed +... 14%, 2048 KB, 3556 KB/s, 0 seconds passed +... 14%, 2080 KB, 3572 KB/s, 0 seconds passed +... 15%, 2112 KB, 3485 KB/s, 0 seconds passed +... 15%, 2144 KB, 3533 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 15%, 2176 KB, 3572 KB/s, 0 seconds passed +... 15%, 2208 KB, 3592 KB/s, 0 seconds passed +... 16%, 2240 KB, 3509 KB/s, 0 seconds passed +... 16%, 2272 KB, 3552 KB/s, 0 seconds passed +... 16%, 2304 KB, 3592 KB/s, 0 seconds passed +... 16%, 2336 KB, 3610 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 17%, 2368 KB, 3526 KB/s, 0 seconds passed +... 17%, 2400 KB, 3569 KB/s, 0 seconds passed +... 17%, 2432 KB, 3606 KB/s, 0 seconds passed +... 17%, 2464 KB, 3542 KB/s, 0 seconds passed +... 17%, 2496 KB, 3545 KB/s, 0 seconds passed +... 18%, 2528 KB, 3586 KB/s, 0 seconds passed +... 18%, 2560 KB, 3622 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 18%, 2592 KB, 3561 KB/s, 0 seconds passed +... 18%, 2624 KB, 3563 KB/s, 0 seconds passed +... 19%, 2656 KB, 3602 KB/s, 0 seconds passed +... 19%, 2688 KB, 3637 KB/s, 0 seconds passed +... 19%, 2720 KB, 3578 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 19%, 2752 KB, 3579 KB/s, 0 seconds passed +... 20%, 2784 KB, 3616 KB/s, 0 seconds passed +... 20%, 2816 KB, 3650 KB/s, 0 seconds passed +... 20%, 2848 KB, 3590 KB/s, 0 seconds passed +... 20%, 2880 KB, 3592 KB/s, 0 seconds passed +... 20%, 2912 KB, 3629 KB/s, 0 seconds passed +... 21%, 2944 KB, 3661 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 21%, 2976 KB, 3603 KB/s, 0 seconds passed +... 21%, 3008 KB, 3608 KB/s, 0 seconds passed +... 21%, 3040 KB, 3641 KB/s, 0 seconds passed +... 22%, 3072 KB, 3673 KB/s, 0 seconds passed +... 22%, 3104 KB, 3621 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 22%, 3136 KB, 3621 KB/s, 0 seconds passed +... 22%, 3168 KB, 3652 KB/s, 0 seconds passed +... 23%, 3200 KB, 3684 KB/s, 0 seconds passed +... 23%, 3232 KB, 3634 KB/s, 0 seconds passed +... 23%, 3264 KB, 3632 KB/s, 0 seconds passed +... 23%, 3296 KB, 3662 KB/s, 0 seconds passed +... 23%, 3328 KB, 3693 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 24%, 3360 KB, 3643 KB/s, 0 seconds passed +... 24%, 3392 KB, 3641 KB/s, 0 seconds passed +... 24%, 3424 KB, 3671 KB/s, 0 seconds passed +... 24%, 3456 KB, 3703 KB/s, 0 seconds passed +... 25%, 3488 KB, 3655 KB/s, 0 seconds passed +... 25%, 3520 KB, 3654 KB/s, 0 seconds passed +... 25%, 3552 KB, 3681 KB/s, 0 seconds passed +... 25%, 3584 KB, 3711 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 26%, 3616 KB, 3665 KB/s, 0 seconds passed +... 26%, 3648 KB, 3664 KB/s, 0 seconds passed +... 26%, 3680 KB, 3689 KB/s, 0 seconds passed +... 26%, 3712 KB, 3705 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 26%, 3744 KB, 3647 KB/s, 1 seconds passed +... 27%, 3776 KB, 3667 KB/s, 1 seconds passed +... 27%, 3808 KB, 3695 KB/s, 1 seconds passed +... 27%, 3840 KB, 3713 KB/s, 1 seconds passed +... 27%, 3872 KB, 3656 KB/s, 1 seconds passed +... 28%, 3904 KB, 3676 KB/s, 1 seconds passed +... 28%, 3936 KB, 3703 KB/s, 1 seconds passed +... 28%, 3968 KB, 3731 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 28%, 4000 KB, 3684 KB/s, 1 seconds passed +... 29%, 4032 KB, 3684 KB/s, 1 seconds passed +... 29%, 4064 KB, 3710 KB/s, 1 seconds passed +... 29%, 4096 KB, 3737 KB/s, 1 seconds passed +... 29%, 4128 KB, 3690 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 29%, 4160 KB, 3691 KB/s, 1 seconds passed +... 30%, 4192 KB, 3718 KB/s, 1 seconds passed +... 30%, 4224 KB, 3743 KB/s, 1 seconds passed +... 30%, 4256 KB, 3703 KB/s, 1 seconds passed +... 30%, 4288 KB, 3699 KB/s, 1 seconds passed +... 31%, 4320 KB, 3724 KB/s, 1 seconds passed +... 31%, 4352 KB, 3748 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 31%, 4384 KB, 3687 KB/s, 1 seconds passed +... 31%, 4416 KB, 3687 KB/s, 1 seconds passed +... 32%, 4448 KB, 3712 KB/s, 1 seconds passed +... 32%, 4480 KB, 3738 KB/s, 1 seconds passed +... 32%, 4512 KB, 3694 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 32%, 4544 KB, 3693 KB/s, 1 seconds passed +... 32%, 4576 KB, 3718 KB/s, 1 seconds passed +... 33%, 4608 KB, 3743 KB/s, 1 seconds passed +... 33%, 4640 KB, 3702 KB/s, 1 seconds passed +... 33%, 4672 KB, 3700 KB/s, 1 seconds passed +... 33%, 4704 KB, 3723 KB/s, 1 seconds passed +... 34%, 4736 KB, 3747 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 34%, 4768 KB, 3723 KB/s, 1 seconds passed +... 34%, 4800 KB, 3704 KB/s, 1 seconds passed +... 34%, 4832 KB, 3727 KB/s, 1 seconds passed +... 35%, 4864 KB, 3751 KB/s, 1 seconds passed +... 35%, 4896 KB, 3711 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 35%, 4928 KB, 3710 KB/s, 1 seconds passed +... 35%, 4960 KB, 3732 KB/s, 1 seconds passed +... 35%, 4992 KB, 3755 KB/s, 1 seconds passed +... 36%, 5024 KB, 3716 KB/s, 1 seconds passed +... 36%, 5056 KB, 3716 KB/s, 1 seconds passed +... 36%, 5088 KB, 3738 KB/s, 1 seconds passed +... 36%, 5120 KB, 3760 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 37%, 5152 KB, 3721 KB/s, 1 seconds passed +... 37%, 5184 KB, 3722 KB/s, 1 seconds passed +... 37%, 5216 KB, 3742 KB/s, 1 seconds passed +... 37%, 5248 KB, 3764 KB/s, 1 seconds passed +... 38%, 5280 KB, 3726 KB/s, 1 seconds passed +... 38%, 5312 KB, 3726 KB/s, 1 seconds passed +... 38%, 5344 KB, 3746 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 38%, 5376 KB, 3727 KB/s, 1 seconds passed +... 38%, 5408 KB, 3729 KB/s, 1 seconds passed +... 39%, 5440 KB, 3731 KB/s, 1 seconds passed +... 39%, 5472 KB, 3751 KB/s, 1 seconds passed +... 39%, 5504 KB, 3731 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 39%, 5536 KB, 3734 KB/s, 1 seconds passed +... 40%, 5568 KB, 3735 KB/s, 1 seconds passed +... 40%, 5600 KB, 3754 KB/s, 1 seconds passed +... 40%, 5632 KB, 3735 KB/s, 1 seconds passed +... 40%, 5664 KB, 3738 KB/s, 1 seconds passed +... 41%, 5696 KB, 3740 KB/s, 1 seconds passed +... 41%, 5728 KB, 3758 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 41%, 5760 KB, 3739 KB/s, 1 seconds passed +... 41%, 5792 KB, 3745 KB/s, 1 seconds passed +... 41%, 5824 KB, 3745 KB/s, 1 seconds passed +... 42%, 5856 KB, 3763 KB/s, 1 seconds passed +... 42%, 5888 KB, 3781 KB/s, 1 seconds passed +... 42%, 5920 KB, 3749 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 42%, 5952 KB, 3748 KB/s, 1 seconds passed +... 43%, 5984 KB, 3767 KB/s, 1 seconds passed +... 43%, 6016 KB, 3784 KB/s, 1 seconds passed +... 43%, 6048 KB, 3754 KB/s, 1 seconds passed +... 43%, 6080 KB, 3754 KB/s, 1 seconds passed +... 44%, 6112 KB, 3770 KB/s, 1 seconds passed +... 44%, 6144 KB, 3787 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 44%, 6176 KB, 3757 KB/s, 1 seconds passed +... 44%, 6208 KB, 3755 KB/s, 1 seconds passed +... 44%, 6240 KB, 3772 KB/s, 1 seconds passed +... 45%, 6272 KB, 3756 KB/s, 1 seconds passed +... 45%, 6304 KB, 3758 KB/s, 1 seconds passed +... 45%, 6336 KB, 3758 KB/s, 1 seconds passed +... 45%, 6368 KB, 3776 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 46%, 6400 KB, 3759 KB/s, 1 seconds passed +... 46%, 6432 KB, 3761 KB/s, 1 seconds passed +... 46%, 6464 KB, 3762 KB/s, 1 seconds passed +... 46%, 6496 KB, 3779 KB/s, 1 seconds passed +... 47%, 6528 KB, 3762 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 47%, 6560 KB, 3764 KB/s, 1 seconds passed +... 47%, 6592 KB, 3764 KB/s, 1 seconds passed +... 47%, 6624 KB, 3781 KB/s, 1 seconds passed +... 47%, 6656 KB, 3766 KB/s, 1 seconds passed +... 48%, 6688 KB, 3766 KB/s, 1 seconds passed +... 48%, 6720 KB, 3768 KB/s, 1 seconds passed +... 48%, 6752 KB, 3784 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 48%, 6784 KB, 3767 KB/s, 1 seconds passed +... 49%, 6816 KB, 3769 KB/s, 1 seconds passed +... 49%, 6848 KB, 3771 KB/s, 1 seconds passed +... 49%, 6880 KB, 3787 KB/s, 1 seconds passed +... 49%, 6912 KB, 3771 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 50%, 6944 KB, 3771 KB/s, 1 seconds passed +... 50%, 6976 KB, 3773 KB/s, 1 seconds passed +... 50%, 7008 KB, 3789 KB/s, 1 seconds passed +... 50%, 7040 KB, 3773 KB/s, 1 seconds passed +... 50%, 7072 KB, 3774 KB/s, 1 seconds passed +... 51%, 7104 KB, 3779 KB/s, 1 seconds passed +... 51%, 7136 KB, 3792 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 51%, 7168 KB, 3779 KB/s, 1 seconds passed +... 51%, 7200 KB, 3781 KB/s, 1 seconds passed +... 52%, 7232 KB, 3781 KB/s, 1 seconds passed +... 52%, 7264 KB, 3794 KB/s, 1 seconds passed +... 52%, 7296 KB, 3782 KB/s, 1 seconds passed +... 52%, 7328 KB, 3785 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 53%, 7360 KB, 3784 KB/s, 1 seconds passed +... 53%, 7392 KB, 3797 KB/s, 1 seconds passed +... 53%, 7424 KB, 3785 KB/s, 1 seconds passed +... 53%, 7456 KB, 3787 KB/s, 1 seconds passed +... 53%, 7488 KB, 3784 KB/s, 1 seconds passed +... 54%, 7520 KB, 3799 KB/s, 1 seconds passed + +.. parsed-literal:: + + ... 54%, 7552 KB, 3784 KB/s, 1 seconds passed +... 54%, 7584 KB, 3785 KB/s, 2 seconds passed +... 54%, 7616 KB, 3786 KB/s, 2 seconds passed +... 55%, 7648 KB, 3801 KB/s, 2 seconds passed +... 55%, 7680 KB, 3786 KB/s, 2 seconds passed +... 55%, 7712 KB, 3787 KB/s, 2 seconds passed +... 55%, 7744 KB, 3791 KB/s, 2 seconds passed +... 56%, 7776 KB, 3803 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 56%, 7808 KB, 3792 KB/s, 2 seconds passed +... 56%, 7840 KB, 3790 KB/s, 2 seconds passed +... 56%, 7872 KB, 3792 KB/s, 2 seconds passed +... 56%, 7904 KB, 3805 KB/s, 2 seconds passed +... 57%, 7936 KB, 3795 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 57%, 7968 KB, 3792 KB/s, 2 seconds passed +... 57%, 8000 KB, 3796 KB/s, 2 seconds passed +... 57%, 8032 KB, 3807 KB/s, 2 seconds passed +... 58%, 8064 KB, 3797 KB/s, 2 seconds passed +... 58%, 8096 KB, 3799 KB/s, 2 seconds passed +... 58%, 8128 KB, 3798 KB/s, 2 seconds passed +... 58%, 8160 KB, 3809 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 59%, 8192 KB, 3799 KB/s, 2 seconds passed +... 59%, 8224 KB, 3800 KB/s, 2 seconds passed +... 59%, 8256 KB, 3798 KB/s, 2 seconds passed +... 59%, 8288 KB, 3811 KB/s, 2 seconds passed +... 59%, 8320 KB, 3793 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 60%, 8352 KB, 3793 KB/s, 2 seconds passed +... 60%, 8384 KB, 3799 KB/s, 2 seconds passed +... 60%, 8416 KB, 3812 KB/s, 2 seconds passed +... 60%, 8448 KB, 3795 KB/s, 2 seconds passed +... 61%, 8480 KB, 3800 KB/s, 2 seconds passed +... 61%, 8512 KB, 3803 KB/s, 2 seconds passed +... 61%, 8544 KB, 3815 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 61%, 8576 KB, 3803 KB/s, 2 seconds passed +... 62%, 8608 KB, 3802 KB/s, 2 seconds passed +... 62%, 8640 KB, 3804 KB/s, 2 seconds passed +... 62%, 8672 KB, 3816 KB/s, 2 seconds passed +... 62%, 8704 KB, 3805 KB/s, 2 seconds passed +... 62%, 8736 KB, 3805 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 63%, 8768 KB, 3806 KB/s, 2 seconds passed +... 63%, 8800 KB, 3818 KB/s, 2 seconds passed +... 63%, 8832 KB, 3806 KB/s, 2 seconds passed +... 63%, 8864 KB, 3802 KB/s, 2 seconds passed +... 64%, 8896 KB, 3807 KB/s, 2 seconds passed +... 64%, 8928 KB, 3820 KB/s, 2 seconds passed +... 64%, 8960 KB, 3808 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 64%, 8992 KB, 3804 KB/s, 2 seconds passed +... 65%, 9024 KB, 3808 KB/s, 2 seconds passed +... 65%, 9056 KB, 3821 KB/s, 2 seconds passed +... 65%, 9088 KB, 3809 KB/s, 2 seconds passed +... 65%, 9120 KB, 3806 KB/s, 2 seconds passed +... 65%, 9152 KB, 3810 KB/s, 2 seconds passed +... 66%, 9184 KB, 3822 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 66%, 9216 KB, 3810 KB/s, 2 seconds passed +... 66%, 9248 KB, 3808 KB/s, 2 seconds passed +... 66%, 9280 KB, 3811 KB/s, 2 seconds passed +... 67%, 9312 KB, 3823 KB/s, 2 seconds passed +... 67%, 9344 KB, 3812 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 67%, 9376 KB, 3809 KB/s, 2 seconds passed +... 67%, 9408 KB, 3812 KB/s, 2 seconds passed +... 68%, 9440 KB, 3824 KB/s, 2 seconds passed +... 68%, 9472 KB, 3810 KB/s, 2 seconds passed +... 68%, 9504 KB, 3816 KB/s, 2 seconds passed +... 68%, 9536 KB, 3815 KB/s, 2 seconds passed +... 68%, 9568 KB, 3826 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 69%, 9600 KB, 3816 KB/s, 2 seconds passed +... 69%, 9632 KB, 3813 KB/s, 2 seconds passed +... 69%, 9664 KB, 3817 KB/s, 2 seconds passed +... 69%, 9696 KB, 3827 KB/s, 2 seconds passed +... 70%, 9728 KB, 3817 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 70%, 9760 KB, 3812 KB/s, 2 seconds passed +... 70%, 9792 KB, 3817 KB/s, 2 seconds passed +... 70%, 9824 KB, 3828 KB/s, 2 seconds passed +... 71%, 9856 KB, 3815 KB/s, 2 seconds passed +... 71%, 9888 KB, 3815 KB/s, 2 seconds passed +... 71%, 9920 KB, 3819 KB/s, 2 seconds passed +... 71%, 9952 KB, 3830 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 71%, 9984 KB, 3819 KB/s, 2 seconds passed +... 72%, 10016 KB, 3818 KB/s, 2 seconds passed +... 72%, 10048 KB, 3820 KB/s, 2 seconds passed +... 72%, 10080 KB, 3831 KB/s, 2 seconds passed +... 72%, 10112 KB, 3817 KB/s, 2 seconds passed +... 73%, 10144 KB, 3816 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 73%, 10176 KB, 3821 KB/s, 2 seconds passed +... 73%, 10208 KB, 3832 KB/s, 2 seconds passed +... 73%, 10240 KB, 3819 KB/s, 2 seconds passed +... 74%, 10272 KB, 3819 KB/s, 2 seconds passed +... 74%, 10304 KB, 3823 KB/s, 2 seconds passed +... 74%, 10336 KB, 3833 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 74%, 10368 KB, 3821 KB/s, 2 seconds passed +... 74%, 10400 KB, 3821 KB/s, 2 seconds passed +... 75%, 10432 KB, 3824 KB/s, 2 seconds passed +... 75%, 10464 KB, 3834 KB/s, 2 seconds passed +... 75%, 10496 KB, 3822 KB/s, 2 seconds passed +... 75%, 10528 KB, 3822 KB/s, 2 seconds passed +... 76%, 10560 KB, 3825 KB/s, 2 seconds passed +... 76%, 10592 KB, 3836 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 76%, 10624 KB, 3824 KB/s, 2 seconds passed +... 76%, 10656 KB, 3824 KB/s, 2 seconds passed +... 77%, 10688 KB, 3828 KB/s, 2 seconds passed +... 77%, 10720 KB, 3837 KB/s, 2 seconds passed +... 77%, 10752 KB, 3825 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 77%, 10784 KB, 3825 KB/s, 2 seconds passed +... 77%, 10816 KB, 3829 KB/s, 2 seconds passed +... 78%, 10848 KB, 3838 KB/s, 2 seconds passed +... 78%, 10880 KB, 3826 KB/s, 2 seconds passed +... 78%, 10912 KB, 3826 KB/s, 2 seconds passed +... 78%, 10944 KB, 3830 KB/s, 2 seconds passed +... 79%, 10976 KB, 3839 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 79%, 11008 KB, 3828 KB/s, 2 seconds passed +... 79%, 11040 KB, 3828 KB/s, 2 seconds passed +... 79%, 11072 KB, 3831 KB/s, 2 seconds passed +... 80%, 11104 KB, 3840 KB/s, 2 seconds passed +... 80%, 11136 KB, 3828 KB/s, 2 seconds passed +... 80%, 11168 KB, 3829 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 80%, 11200 KB, 3832 KB/s, 2 seconds passed +... 80%, 11232 KB, 3833 KB/s, 2 seconds passed +... 81%, 11264 KB, 3830 KB/s, 2 seconds passed +... 81%, 11296 KB, 3829 KB/s, 2 seconds passed +... 81%, 11328 KB, 3833 KB/s, 2 seconds passed +... 81%, 11360 KB, 3834 KB/s, 2 seconds passed + +.. parsed-literal:: + + ... 82%, 11392 KB, 3831 KB/s, 2 seconds passed +... 82%, 11424 KB, 3831 KB/s, 2 seconds passed +... 82%, 11456 KB, 3833 KB/s, 2 seconds passed +... 82%, 11488 KB, 3834 KB/s, 2 seconds passed +... 82%, 11520 KB, 3832 KB/s, 3 seconds passed +... 83%, 11552 KB, 3832 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 83%, 11584 KB, 3834 KB/s, 3 seconds passed +... 83%, 11616 KB, 3794 KB/s, 3 seconds passed +... 83%, 11648 KB, 3804 KB/s, 3 seconds passed +... 84%, 11680 KB, 3814 KB/s, 3 seconds passed +... 84%, 11712 KB, 3824 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 84%, 11744 KB, 3796 KB/s, 3 seconds passed +... 84%, 11776 KB, 3806 KB/s, 3 seconds passed +... 85%, 11808 KB, 3815 KB/s, 3 seconds passed +... 85%, 11840 KB, 3825 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 85%, 11872 KB, 3798 KB/s, 3 seconds passed +... 85%, 11904 KB, 3807 KB/s, 3 seconds passed +... 85%, 11936 KB, 3817 KB/s, 3 seconds passed +... 86%, 11968 KB, 3826 KB/s, 3 seconds passed +... 86%, 12000 KB, 3798 KB/s, 3 seconds passed +... 86%, 12032 KB, 3808 KB/s, 3 seconds passed +... 86%, 12064 KB, 3818 KB/s, 3 seconds passed +... 87%, 12096 KB, 3827 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 87%, 12128 KB, 3800 KB/s, 3 seconds passed +... 87%, 12160 KB, 3809 KB/s, 3 seconds passed +... 87%, 12192 KB, 3819 KB/s, 3 seconds passed +... 88%, 12224 KB, 3827 KB/s, 3 seconds passed +... 88%, 12256 KB, 3801 KB/s, 3 seconds passed +... 88%, 12288 KB, 3810 KB/s, 3 seconds passed +... 88%, 12320 KB, 3820 KB/s, 3 seconds passed +... 88%, 12352 KB, 3829 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 89%, 12384 KB, 3802 KB/s, 3 seconds passed +... 89%, 12416 KB, 3812 KB/s, 3 seconds passed +... 89%, 12448 KB, 3821 KB/s, 3 seconds passed +... 89%, 12480 KB, 3830 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 90%, 12512 KB, 3804 KB/s, 3 seconds passed +... 90%, 12544 KB, 3813 KB/s, 3 seconds passed +... 90%, 12576 KB, 3822 KB/s, 3 seconds passed +... 90%, 12608 KB, 3831 KB/s, 3 seconds passed +... 91%, 12640 KB, 3805 KB/s, 3 seconds passed +... 91%, 12672 KB, 3814 KB/s, 3 seconds passed +... 91%, 12704 KB, 3823 KB/s, 3 seconds passed +... 91%, 12736 KB, 3832 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 91%, 12768 KB, 3807 KB/s, 3 seconds passed +... 92%, 12800 KB, 3815 KB/s, 3 seconds passed +... 92%, 12832 KB, 3824 KB/s, 3 seconds passed +... 92%, 12864 KB, 3833 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 92%, 12896 KB, 3808 KB/s, 3 seconds passed +... 93%, 12928 KB, 3816 KB/s, 3 seconds passed +... 93%, 12960 KB, 3825 KB/s, 3 seconds passed +... 93%, 12992 KB, 3834 KB/s, 3 seconds passed +... 93%, 13024 KB, 3809 KB/s, 3 seconds passed +... 94%, 13056 KB, 3817 KB/s, 3 seconds passed +... 94%, 13088 KB, 3826 KB/s, 3 seconds passed +... 94%, 13120 KB, 3835 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 94%, 13152 KB, 3810 KB/s, 3 seconds passed +... 94%, 13184 KB, 3818 KB/s, 3 seconds passed +... 95%, 13216 KB, 3827 KB/s, 3 seconds passed +... 95%, 13248 KB, 3836 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 95%, 13280 KB, 3811 KB/s, 3 seconds passed +... 95%, 13312 KB, 3819 KB/s, 3 seconds passed +... 96%, 13344 KB, 3827 KB/s, 3 seconds passed +... 96%, 13376 KB, 3836 KB/s, 3 seconds passed +... 96%, 13408 KB, 3811 KB/s, 3 seconds passed +... 96%, 13440 KB, 3819 KB/s, 3 seconds passed +... 97%, 13472 KB, 3828 KB/s, 3 seconds passed +... 97%, 13504 KB, 3837 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 97%, 13536 KB, 3812 KB/s, 3 seconds passed +... 97%, 13568 KB, 3821 KB/s, 3 seconds passed +... 97%, 13600 KB, 3829 KB/s, 3 seconds passed +... 98%, 13632 KB, 3837 KB/s, 3 seconds passed +... 98%, 13664 KB, 3813 KB/s, 3 seconds passed +... 98%, 13696 KB, 3822 KB/s, 3 seconds passed +... 98%, 13728 KB, 3830 KB/s, 3 seconds passed +... 99%, 13760 KB, 3838 KB/s, 3 seconds passed + +.. parsed-literal:: + + ... 99%, 13792 KB, 3815 KB/s, 3 seconds passed +... 99%, 13824 KB, 3822 KB/s, 3 seconds passed +... 99%, 13856 KB, 3831 KB/s, 3 seconds passed +... 100%, 13879 KB, 3837 KB/s, 3 seconds passed + + + + +Convert a Model to OpenVINO IR format +------------------------------------- + + Specify, display and run the Model Converter command to convert the model to OpenVINO IR format. Model conversion may take a while. The @@ -190,7 +851,7 @@ that are already in OpenVINO IR format, conversion will be skipped. .. code:: ipython3 ## Uncomment the next line to show Help in omz_converter which explains the command-line options. - + # !omz_converter --help .. code:: ipython3 @@ -198,7 +859,7 @@ that are already in OpenVINO IR format, conversion will be skipped. convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}" display(Markdown(f"Convert command: `{convert_command}`")) display(Markdown(f"Converting {model_name}...")) - + ! $convert_command @@ -214,25 +875,43 @@ Converting mobilenet-v2-pytorch… .. parsed-literal:: ========== Converting mobilenet-v2-pytorch to ONNX - Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob - + Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob + + + +.. parsed-literal:: + ONNX check passed successfully. - + + +.. parsed-literal:: + + ========== Converting mobilenet-v2-pytorch to IR (FP16) - Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=model/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True - + Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=model/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True + + + +.. parsed-literal:: + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin - + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin + + + + + +Get Model Information +--------------------- -Get Model Information ---------------------------------------------------------------- The Info Dumper prints the following information for Open Model Zoo models: @@ -256,7 +935,7 @@ information in a dictionary. model_info_output = %sx omz_info_dumper --name $model_name model_info = json.loads(model_info_output.get_nlstr()) - + if len(model_info) > 1: NotebookAlert( f"There are multiple IR files for the {model_name} model. The first model in the " @@ -264,7 +943,7 @@ information in a dictionary. "`selected_model_info` in the cell below to select a different model from the list.", "warning", ) - + model_info @@ -277,8 +956,8 @@ information in a dictionary. 'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" .\nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.\nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.', 'framework': 'pytorch', 'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE', - 'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/accuracy-check.yml', - 'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/model.yml', + 'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/accuracy-check.yml', + 'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/model.yml', 'precisions': ['FP16', 'FP32'], 'quantization_output_precisions': ['FP16-INT8', 'FP32-INT8'], 'subdirectory': 'public/mobilenet-v2-pytorch', @@ -310,8 +989,10 @@ file. model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True -Run Benchmark Tool ------------------------------------------------------------- +Run Benchmark Tool +------------------ + + By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per @@ -327,7 +1008,7 @@ image) and throughput values (frames per second). benchmark_command = f"benchmark_app -m {model_path} -t 15" display(Markdown(f"Benchmark command: `{benchmark_command}`")) display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds...")) - + ! $benchmark_command @@ -347,18 +1028,22 @@ seconds… [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.1.0-12185-9e6b00e51cd-releases/2023/1 - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: + + +.. parsed-literal:: + [ INFO ] CPU - [ INFO ] Build ................................. 2023.1.0-12185-9e6b00e51cd-releases/2023/1 - [ INFO ] - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 30.99 ms + [ INFO ] Read model took 30.88 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] data (node: data) : f32 / [N,C,H,W] / [1,3,224,224] @@ -372,7 +1057,11 @@ seconds… [ INFO ] Model outputs: [ INFO ] prob (node: prob) : f32 / [...] / [1,1000] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 148.02 ms + + +.. parsed-literal:: + + [ INFO ] Compile model took 154.97 ms [Step 8/11] Querying optimal runtime parameters [ INFO ] Model: [ INFO ] NETWORK_NAME: main_graph @@ -380,9 +1069,9 @@ seconds… [ INFO ] NUM_STREAMS: 6 [ INFO ] AFFINITY: Affinity.CORE [ INFO ] INFERENCE_NUM_THREADS: 24 - [ INFO ] PERF_COUNT: False + [ INFO ] PERF_COUNT: NO [ INFO ] INFERENCE_PRECISION_HINT: - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT + [ INFO ] PERFORMANCE_HINT: THROUGHPUT [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 [ INFO ] ENABLE_CPU_PINNING: True @@ -393,24 +1082,34 @@ seconds… [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0 [Step 9/11] Creating infer requests and preparing input tensors [ WARNING ] No input files were given for input 'data'!. This input will be filled with random values! - [ INFO ] Fill input 'data' with random values + [ INFO ] Fill input 'data' with random values [Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration) [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 6.55 ms + + +.. parsed-literal:: + + [ INFO ] First inference took 6.26 ms + + +.. parsed-literal:: + [Step 11/11] Dumping statistics report [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 20166 iterations - [ INFO ] Duration: 15006.87 ms + [ INFO ] Count: 20028 iterations + [ INFO ] Duration: 15005.22 ms [ INFO ] Latency: - [ INFO ] Median: 4.32 ms - [ INFO ] Average: 4.32 ms - [ INFO ] Min: 2.50 ms - [ INFO ] Max: 12.20 ms - [ INFO ] Throughput: 1343.78 FPS + [ INFO ] Median: 4.36 ms + [ INFO ] Average: 4.37 ms + [ INFO ] Min: 2.74 ms + [ INFO ] Max: 12.78 ms + [ INFO ] Throughput: 1334.74 FPS + + +Benchmark with Different Settings +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Benchmark with Different Settings -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``benchmark_app`` tool displays logging information that is not always necessary. A more compact result is achieved when the output is @@ -454,7 +1153,7 @@ the cell below that, you display available devices on the system. benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}" display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**")) display(Markdown(f"Benchmark command: `{benchmark_command}`")) - + benchmark_output = %sx $benchmark_command print("command ended") benchmark_result = [line for line in benchmark_output @@ -464,7 +1163,7 @@ the cell below that, you display available devices on the system. .. code:: ipython3 core = ov.Core() - + # Show devices available for OpenVINO Runtime for device in core.available_devices: device_name = core.get_property(device, "FULL_DEVICE_NAME") @@ -494,7 +1193,7 @@ Benchmark command: .. parsed-literal:: command ended - + .. code:: ipython3 @@ -515,7 +1214,7 @@ Benchmark command: .. parsed-literal:: command ended - + .. code:: ipython3 diff --git a/docs/notebooks/105-language-quantize-bert-with-output.rst b/docs/notebooks/105-language-quantize-bert-with-output.rst index 25b2d7c0098..295fde2a3b1 100644 --- a/docs/notebooks/105-language-quantize-bert-with-output.rst +++ b/docs/notebooks/105-language-quantize-bert-with-output.rst @@ -22,8 +22,8 @@ and datasets. It consists of the following steps: - Compare the performance of the original, converted and quantized models. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ - `Settings <#settings>`__ @@ -36,13 +36,13 @@ and datasets. It consists of the following steps: - `Select inference device <#select-inference-device>`__ - `Compare F1-score of FP32 and INT8 - models <#compare-f-score-of-fp-and-int-models>`__ + models <#compare-f1-score-of-fp32-and-int8-models>`__ - `Compare Performance of the Original, Converted and Quantized Models <#compare-performance-of-the-original-converted-and-quantized-models>`__ .. code:: ipython3 - %pip install -q "nncf>=2.5.0" + %pip install -q "nncf>=2.5.0" %pip install -q "transformers" datasets evaluate --extra-index-url https://download.pytorch.org/whl/cpu %pip install -q "openvino>=2023.1.0" @@ -50,7 +50,15 @@ and datasets. It consists of the following steps: .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. @@ -67,7 +75,7 @@ Imports from zipfile import ZipFile from typing import Iterable from typing import Any - + import datasets import evaluate import numpy as np @@ -76,7 +84,7 @@ Imports import openvino as ov import torch from transformers import BertForSequenceClassification, BertTokenizer - + # Fetch `notebook_utils` module import urllib.request urllib.request.urlretrieve( @@ -88,10 +96,14 @@ Imports .. parsed-literal:: - 2023-12-06 22:34:55.977192: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-12-06 22:34:56.010680: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + 2024-01-25 22:37:51.906403: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 22:37:51.939998: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-12-06 22:34:56.639162: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + 2024-01-25 22:37:52.576201: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT .. parsed-literal:: @@ -112,7 +124,7 @@ Settings MODEL_LINK = "https://download.pytorch.org/tutorial/MRPC.zip" FILE_NAME = MODEL_LINK.split("/")[-1] PRETRAINED_MODEL_DIR = os.path.join(MODEL_DIR, "MRPC") - + os.makedirs(DATA_DIR, exist_ok=True) os.makedirs(MODEL_DIR, exist_ok=True) @@ -157,10 +169,10 @@ PyTorch model formats are supported: input_shape = ov.PartialShape([1, -1]) ir_model_xml = Path(MODEL_DIR) / "bert_mrpc.xml" core = ov.Core() - + torch_model = BertForSequenceClassification.from_pretrained(PRETRAINED_MODEL_DIR) torch_model.eval - + input_info = [("input_ids", input_shape, np.int64),("attention_mask", input_shape, np.int64),("token_type_ids", input_shape, np.int64)] default_input = torch.ones(1, MAX_SEQ_LENGTH, dtype=torch.int64) inputs = { @@ -168,7 +180,7 @@ PyTorch model formats are supported: "attention_mask": default_input, "token_type_ids": default_input, } - + # Convert the PyTorch model to OpenVINO IR FP32. if not ir_model_xml.exists(): model = ov.convert_model(torch_model, example_input=inputs, input=input_info) @@ -177,6 +189,12 @@ PyTorch model formats are supported: model = core.read_model(ir_model_xml) +.. parsed-literal:: + + /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() + + .. parsed-literal:: WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11. @@ -185,6 +203,15 @@ PyTorch model formats are supported: .. parsed-literal:: [ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s. + + +.. parsed-literal:: + + WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.1.0+cpu. If you encounter issues, consider switching to torch==2.1.2 + + +.. parsed-literal:: + No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' @@ -203,16 +230,16 @@ tokenizer from HuggingFace. def create_data_source(): raw_dataset = datasets.load_dataset('glue', 'mrpc', split='validation') tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_DIR) - + def _preprocess_fn(examples): texts = (examples['sentence1'], examples['sentence2']) result = tokenizer(*texts, padding='max_length', max_length=MAX_SEQ_LENGTH, truncation=True) result['labels'] = examples['label'] return result processed_dataset = raw_dataset.map(_preprocess_fn, batched=True, batch_size=1) - + return processed_dataset - + data_source = create_data_source() Optimize model using NNCF Post-training Quantization API @@ -234,7 +261,7 @@ The optimization process contains the following steps: .. code:: ipython3 INPUT_NAMES = [key for key in inputs.keys()] - + def transform_fn(data_item): """ Extract the model's input from the data item. @@ -245,7 +272,7 @@ The optimization process contains the following steps: name: np.asarray([data_item[name]], dtype=np.int64) for name in INPUT_NAMES } return inputs - + calibration_dataset = nncf.Dataset(data_source, transform_fn) # Quantize the model. By specifying model_type, we specify additional transformer patterns in the model. quantized_model = nncf.quantize(model, calibration_dataset, @@ -266,10 +293,6 @@ The optimization process contains the following steps: -.. raw:: html - -
-    
@@ -287,11 +310,6 @@ The optimization process contains the following steps: -.. raw:: html - -
-    
- .. parsed-literal:: @@ -299,6 +317,11 @@ The optimization process contains the following steps: INFO:nncf:36 ignored nodes were found by name in the NNCFGraph +.. parsed-literal:: + + INFO:nncf:50 ignored nodes were found by name in the NNCFGraph + + .. parsed-literal:: @@ -313,10 +336,6 @@ The optimization process contains the following steps: -.. raw:: html - -
-    
@@ -334,10 +353,6 @@ The optimization process contains the following steps: -.. raw:: html - -
-    
@@ -368,14 +383,14 @@ select device from dropdown list for running inference using OpenVINO .. code:: ipython3 import ipywidgets as widgets - + device = widgets.Dropdown( options=core.available_devices + ["AUTO"], value='AUTO', description='Device:', disabled=False, ) - + device @@ -403,10 +418,10 @@ changing ``sample_idx`` to another value (from 0 to 407). sample_idx = 5 sample = data_source[sample_idx] inputs = {k: torch.unsqueeze(torch.tensor(sample[k]), 0) for k in ['input_ids', 'token_type_ids', 'attention_mask']} - + result = compiled_quantized_model(inputs)[output_layer] result = np.argmax(result) - + print(f"Text 1: {sample['sentence1']}") print(f"Text 2: {sample['sentence2']}") print(f"The same meaning: {'yes' if result == 1 else 'no'}") @@ -428,12 +443,12 @@ Compare F1-score of FP32 and INT8 models def validate(model: ov.Model, dataset: Iterable[Any]) -> float: """ - Evaluate the model on GLUE dataset. + Evaluate the model on GLUE dataset. Returns F1 score metric. """ compiled_model = core.compile_model(model, device_name=device.value) output_layer = compiled_model.output(0) - + metric = evaluate.load('glue', 'mrpc') for batch in dataset: inputs = [ @@ -444,14 +459,14 @@ Compare F1-score of FP32 and INT8 models metric.add_batch(predictions=[predictions], references=[batch['labels']]) metrics = metric.compute() f1_score = metrics['f1'] - + return f1_score - - + + print('Checking the accuracy of the original model:') metric = validate(model, data_source) print(f'F1 score: {metric:.4f}') - + print('Checking the accuracy of the quantized model:') metric = validate(quantized_model, data_source) print(f'F1 score: {metric:.4f}') @@ -460,8 +475,16 @@ Compare F1-score of FP32 and INT8 models .. parsed-literal:: Checking the accuracy of the original model: + + +.. parsed-literal:: + F1 score: 0.9019 Checking the accuracy of the quantized model: + + +.. parsed-literal:: + F1 score: 0.8969 @@ -485,7 +508,7 @@ Frames Per Second (FPS) for images. num_samples = 50 sample = data_source[0] inputs = {k: torch.unsqueeze(torch.tensor(sample[k]), 0) for k in ['input_ids', 'token_type_ids', 'attention_mask']} - + with torch.no_grad(): start = time.perf_counter() for _ in range(num_samples): @@ -496,7 +519,7 @@ Frames Per Second (FPS) for images. f"PyTorch model on CPU: {time_torch / num_samples:.3f} seconds per sentence, " f"SPS: {num_samples / time_torch:.2f}" ) - + start = time.perf_counter() for _ in range(num_samples): compiled_model(inputs) @@ -506,7 +529,7 @@ Frames Per Second (FPS) for images. f"IR FP32 model in OpenVINO Runtime/{device.value}: {time_ir / num_samples:.3f} " f"seconds per sentence, SPS: {num_samples / time_ir:.2f}" ) - + start = time.perf_counter() for _ in range(num_samples): compiled_quantized_model(inputs) @@ -525,9 +548,17 @@ Frames Per Second (FPS) for images. .. parsed-literal:: - PyTorch model on CPU: 0.073 seconds per sentence, SPS: 13.77 - IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 48.61 - OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 109.06 + PyTorch model on CPU: 0.074 seconds per sentence, SPS: 13.56 + + +.. parsed-literal:: + + IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 48.16 + + +.. parsed-literal:: + + OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 109.03 Finally, measure the inference performance of OpenVINO ``FP32`` and @@ -557,24 +588,24 @@ in OpenVINO. [Step 2/11] Loading OpenVINO Runtime [ WARNING ] Default duration 120 seconds is used for unknown device device.value [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: - [ INFO ] - [ INFO ] + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration - [ ERROR ] Exception from src/inference/src/core.cpp:244: - Exception from src/inference/src/dev/core_impl.cpp:559: + [ ERROR ] Exception from src/inference/src/core.cpp:228: + Exception from src/inference/src/dev/core_impl.cpp:560: Device with "device" name is not registered in the OpenVINO Runtime - + Traceback (most recent call last): - File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 165, in main + File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main supported_properties = benchmark.core.get_property(device, properties.supported_properties()) - RuntimeError: Exception from src/inference/src/core.cpp:244: - Exception from src/inference/src/dev/core_impl.cpp:559: + RuntimeError: Exception from src/inference/src/core.cpp:228: + Exception from src/inference/src/dev/core_impl.cpp:560: Device with "device" name is not registered in the OpenVINO Runtime - - + + .. code:: ipython3 @@ -590,22 +621,22 @@ in OpenVINO. [Step 2/11] Loading OpenVINO Runtime [ WARNING ] Default duration 120 seconds is used for unknown device device.value [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: - [ INFO ] - [ INFO ] + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration - [ ERROR ] Exception from src/inference/src/core.cpp:244: - Exception from src/inference/src/dev/core_impl.cpp:559: + [ ERROR ] Exception from src/inference/src/core.cpp:228: + Exception from src/inference/src/dev/core_impl.cpp:560: Device with "device" name is not registered in the OpenVINO Runtime - - Traceback (most recent call last): - File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 165, in main - supported_properties = benchmark.core.get_property(device, properties.supported_properties()) - RuntimeError: Exception from src/inference/src/core.cpp:244: - Exception from src/inference/src/dev/core_impl.cpp:559: - Device with "device" name is not registered in the OpenVINO Runtime - - + + Traceback (most recent call last): + File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main + supported_properties = benchmark.core.get_property(device, properties.supported_properties()) + RuntimeError: Exception from src/inference/src/core.cpp:228: + Exception from src/inference/src/dev/core_impl.cpp:560: + Device with "device" name is not registered in the OpenVINO Runtime + + diff --git a/docs/notebooks/106-auto-device-with-output.rst b/docs/notebooks/106-auto-device-with-output.rst index ec3dc7014b8..970b8d7cfe0 100644 --- a/docs/notebooks/106-auto-device-with-output.rst +++ b/docs/notebooks/106-auto-device-with-output.rst @@ -13,7 +13,7 @@ network efficiently. Next, if dedicated accelerators are available, these devices are preferred (for example, integrated and discrete -`GPU `__). +`GPU `__). `CPU `__ is used as the default “fallback device”. Keep in mind that AUTO makes this selection only once, during the loading of a model. @@ -30,15 +30,13 @@ first inference. auto -**Table of contents:** +Table of contents: +^^^^^^^^^^^^^^^^^^ - -- `Import modules and create - Core <#import-modules-and-create-core>`__ +- `Import modules and create Core <#import-modules-and-create-core>`__ - `Convert the model to OpenVINO IR format <#convert-the-model-to-openvino-ir-format>`__ -- `(1) Simplify selection - logic <#-simplify-selection-logic>`__ +- `(1) Simplify selection logic <#1-simplify-selection-logic>`__ - `Default behavior of Core::compile_model API without device_name <#default-behavior-of-corecompile_model-api-without-device_name>`__ @@ -46,7 +44,7 @@ first inference. API <#explicitly-pass-auto-as-device_name-to-corecompile_model-api>`__ - `(2) Improve the first inference - latency <#-improve-the-first-inference-latency>`__ + latency <#2-improve-the-first-inference-latency>`__ - `Load an Image <#load-an-image>`__ - `Load the model to GPU device and perform @@ -55,19 +53,18 @@ first inference. inference <#load-the-model-using-auto-device-and-do-inference>`__ - `(3) Achieve different performance for different - targets <#-achieve-different-performance-for-different-targets>`__ + targets <#3-achieve-different-performance-for-different-targets>`__ - - `Class and callback - definition <#class-and-callback-definition>`__ + - `Class and callback definition <#class-and-callback-definition>`__ - `Inference with THROUGHPUT hint <#inference-with-throughput-hint>`__ - - `Inference with LATENCY - hint <#inference-with-latency-hint>`__ - - `Difference in FPS and - latency <#difference-in-fps-and-latency>`__ + - `Inference with LATENCY hint <#inference-with-latency-hint>`__ + - `Difference in FPS and latency <#difference-in-fps-and-latency>`__ + +Import modules and create Core +------------------------------ + -Import modules and create Core ------------------------------------------------------------------------- .. code:: ipython3 @@ -102,8 +99,10 @@ Import modules and create Core device to have meaningful results. -Convert the model to OpenVINO IR format ---------------------------------------------------------------------------------- +Convert the model to OpenVINO IR format +--------------------------------------- + + This tutorial uses `resnet50 `__ @@ -148,11 +147,15 @@ For more information about model conversion API, see this IR model saved to model/resnet50.xml -(1) Simplify selection logic ----------------------------------------------------------------------- +(1) Simplify selection logic +---------------------------- + + + +Default behavior of Core::compile_model API without device_name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + -Default behavior of Core::compile_model API without device_name -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, ``compile_model`` API will select **AUTO** as ``device_name`` if no device is specified. @@ -171,14 +174,14 @@ By default, ``compile_model`` API will select **AUTO** as .. parsed-literal:: - [22:37:18.2538]I[plugin.cpp:537][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY - [22:37:18.2539]I[plugin.cpp:537][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0 - [22:37:18.2539]I[plugin.cpp:537][AUTO] device:CPU, config:PERF_COUNT=NO - [22:37:18.2539]I[plugin.cpp:542][AUTO] device:CPU, priority:0 - [22:37:18.2540]I[schedule.cpp:17][AUTO] scheduler starting - [22:37:18.2540]I[auto_schedule.cpp:131][AUTO] select device:CPU - [22:37:18.3716]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished - [22:37:18.3717]I[plugin.cpp:572][AUTO] underlying hardware does not support hardware context + [22:41:12.8903]I[plugin.cpp:536][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY + [22:41:12.8903]I[plugin.cpp:536][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0 + [22:41:12.8904]I[plugin.cpp:536][AUTO] device:CPU, config:PERF_COUNT=NO + [22:41:12.8904]I[plugin.cpp:541][AUTO] device:CPU, priority:0 + [22:41:12.8904]I[schedule.cpp:17][AUTO] scheduler starting + [22:41:12.8904]I[auto_schedule.cpp:131][AUTO] select device:CPU + [22:41:13.0500]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished + [22:41:13.0502]I[plugin.cpp:569][AUTO] underlying hardware does not support hardware context Successfully compiled model without a device_name. @@ -191,12 +194,14 @@ By default, ``compile_model`` API will select **AUTO** as .. parsed-literal:: - [22:37:18.3836]I[schedule.cpp:303][AUTO] scheduler ending Deleted compiled_model + [22:41:13.0623]I[schedule.cpp:303][AUTO] scheduler ending + + +Explicitly pass AUTO as device_name to Core::compile_model API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Explicitly pass AUTO as device_name to Core::compile_model API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is optional, but passing AUTO explicitly as ``device_name`` may improve readability of your code. @@ -229,8 +234,10 @@ improve readability of your code. Deleted compiled_model -(2) Improve the first inference latency ---------------------------------------------------------------------------------- +(2) Improve the first inference latency +--------------------------------------- + + One of the benefits of using AUTO device selection is reducing FIL (first inference latency). FIL is the model compilation time combined @@ -243,8 +250,10 @@ This initialization time may be intolerable for some applications. To avoid this delay, the AUTO uses CPU transparently as the first inference device until GPU is ready. -Load an Image -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Load an Image +~~~~~~~~~~~~~ + + torchvision library provides model specific input transformation function, we will reuse it for preparing input data. @@ -289,8 +298,10 @@ function, we will reuse it for preparing input data. -Load the model to GPU device and perform inference -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Load the model to GPU device and perform inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + .. code:: ipython3 @@ -316,8 +327,10 @@ Load the model to GPU device and perform inference A GPU device is not available. Available devices are: ['CPU'] -Load the model using AUTO device and do inference -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Load the model using AUTO device and do inference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + When GPU is the best available device, the first few inferences will be executed on CPU until GPU is ready. @@ -340,7 +353,7 @@ executed on CPU until GPU is ready. .. parsed-literal:: - Time to load model using AUTO device and get first inference: 0.14 seconds. + Time to load model using AUTO device and get first inference: 0.15 seconds. .. code:: ipython3 @@ -348,8 +361,10 @@ executed on CPU until GPU is ready. # Deleted model will wait for compiling on the selected device to complete. del compiled_model -(3) Achieve different performance for different targets -------------------------------------------------------------------------------------------------- +(3) Achieve different performance for different targets +------------------------------------------------------- + + It is an advantage to define **performance hints** when using Automatic Device Selection. By specifying a **THROUGHPUT** or **LATENCY** hint, @@ -361,13 +376,15 @@ completely portable between devices – meaning AUTO can configure the performance hint on whichever device is being used. For more information, refer to the `Performance -Hints `__ +Hints `__ section of `Automatic Device Selection `__ article. -Class and callback definition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Class and callback definition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + .. code:: ipython3 @@ -465,8 +482,10 @@ Class and callback definition metrics_update_interval = 10 metrics_update_num = 6 -Inference with THROUGHPUT hint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Inference with THROUGHPUT hint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Loop for inference and update the FPS/Latency every @metrics_update_interval seconds. @@ -503,18 +522,52 @@ Loop for inference and update the FPS/Latency every .. parsed-literal:: Compiling Model for AUTO device with THROUGHPUT hint + + +.. parsed-literal:: + Start inference, 6 groups of FPS/latency will be measured over 10s intervals - throughput: 182.05fps, latency: 31.31ms, time interval: 10.01s - throughput: 182.68fps, latency: 32.06ms, time interval: 10.01s - throughput: 183.56fps, latency: 31.93ms, time interval: 10.02s - throughput: 182.77fps, latency: 32.06ms, time interval: 10.01s - throughput: 182.17fps, latency: 32.13ms, time interval: 10.00s - throughput: 182.39fps, latency: 32.15ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 184.87fps, latency: 30.81ms, time interval: 10.01s + + +.. parsed-literal:: + + throughput: 185.21fps, latency: 31.62ms, time interval: 10.02s + + +.. parsed-literal:: + + throughput: 185.47fps, latency: 31.54ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 185.75fps, latency: 31.53ms, time interval: 10.01s + + +.. parsed-literal:: + + throughput: 185.09fps, latency: 31.64ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 184.42fps, latency: 31.76ms, time interval: 10.00s + + +.. parsed-literal:: + Done -Inference with LATENCY hint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Inference with LATENCY hint +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Loop for inference and update the FPS/Latency for each @metrics_update_interval seconds @@ -552,18 +605,52 @@ Loop for inference and update the FPS/Latency for each .. parsed-literal:: Compiling Model for AUTO Device with LATENCY hint + + +.. parsed-literal:: + Start inference, 6 groups fps/latency will be out with 10s interval - throughput: 140.29fps, latency: 6.65ms, time interval: 10.00s - throughput: 142.87fps, latency: 6.63ms, time interval: 10.00s - throughput: 142.41fps, latency: 6.64ms, time interval: 10.01s - throughput: 142.91fps, latency: 6.63ms, time interval: 10.01s - throughput: 142.78fps, latency: 6.64ms, time interval: 10.00s - throughput: 142.80fps, latency: 6.64ms, time interval: 10.01s + + +.. parsed-literal:: + + throughput: 138.57fps, latency: 6.67ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 141.04fps, latency: 6.66ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 140.74fps, latency: 6.66ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 141.60fps, latency: 6.68ms, time interval: 10.01s + + +.. parsed-literal:: + + throughput: 141.73fps, latency: 6.68ms, time interval: 10.00s + + +.. parsed-literal:: + + throughput: 141.45fps, latency: 6.67ms, time interval: 10.00s + + +.. parsed-literal:: + Done -Difference in FPS and latency -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Difference in FPS and latency +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + .. code:: ipython3 diff --git a/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_27_0.png b/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_27_0.png index da915be3e30..619a80e73ac 100644 --- a/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_27_0.png +++ b/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_27_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f838f2aa8318b73e4dadf5d48bbfb1332a4bebc2ca4d854dd81e77c93b5b244 -size 26784 +oid sha256:1b8fb134deae365a3c5b358b032e2ad1e5ccaccae51cb176f250ba946461d8d6 +size 25724 diff --git a/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_28_0.png b/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_28_0.png index 42aa7ec559f..f69a128eb75 100644 --- a/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_28_0.png +++ b/docs/notebooks/106-auto-device-with-output_files/106-auto-device-with-output_28_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8187fa0479ea8f4d92a4b1cb42aa15f5377ce2630fa924bee73168ba2c5d10b4 -size 39984 +oid sha256:d311a41fc8902df2f1ca223ffcdcf1ff634d160376b890b3a571326912b1d88a +size 40017 diff --git a/docs/notebooks/106-auto-device-with-output_files/index.html b/docs/notebooks/106-auto-device-with-output_files/index.html deleted file mode 100644 index ec993324132..00000000000 --- a/docs/notebooks/106-auto-device-with-output_files/index.html +++ /dev/null @@ -1,10 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/106-auto-device-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/106-auto-device-with-output_files/


../
-106-auto-device-with-output_14_1.jpg               31-Oct-2023 00:35              121563
-106-auto-device-with-output_14_1.png               31-Oct-2023 00:35              869661
-106-auto-device-with-output_27_0.png               31-Oct-2023 00:35               26784
-106-auto-device-with-output_28_0.png               31-Oct-2023 00:35               39984
-

- diff --git a/docs/notebooks/106-auto-device-with-output_files/index.html.tmp b/docs/notebooks/106-auto-device-with-output_files/index.html.tmp new file mode 100644 index 00000000000..1329909be0d --- /dev/null +++ b/docs/notebooks/106-auto-device-with-output_files/index.html.tmp @@ -0,0 +1,10 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/106-auto-device-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/106-auto-device-with-output_files/


../
+106-auto-device-with-output_14_1.jpg               26-Jan-2024 01:05              121563
+106-auto-device-with-output_14_1.png               26-Jan-2024 01:05              869661
+106-auto-device-with-output_27_0.png               26-Jan-2024 01:05               25724
+106-auto-device-with-output_28_0.png               26-Jan-2024 01:05               40017
+

+ diff --git a/docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst b/docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst index 0ba82c888af..2ac739f7abf 100644 --- a/docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst +++ b/docs/notebooks/107-speech-recognition-quantization-data2vec-with-output.rst @@ -19,8 +19,8 @@ steps: - Compare performance of the original and quantized models. - Compare Accuracy of the Original and Quantized Models. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Download and prepare model <#download-and-prepare-model>`__ @@ -35,7 +35,7 @@ steps: dataset <#validate-model-accuracy-on-dataset>`__ - `Quantization <#quantization>`__ - `Check INT8 model inference - result <#check-int-model-inference-result>`__ + result <#check-int8-model-inference-result>`__ - `Compare Performance of the Original and Quantized Models <#compare-performance-of-the-original-and-quantized-models>`__ - `Compare Accuracy of the Original and Quantized @@ -79,26 +79,17 @@ model specific pre- and post-processing steps. .. code:: ipython3 - %pip install -q "openvino>=2023.1.0" "nncf>=2.5.0" - %pip install -q datasets "torchmetrics>=0.11.0" --extra-index-url https://download.pytorch.org/whl/cpu - %pip install -q soundfile librosa transformers --extra-index-url https://download.pytorch.org/whl/cpu + %pip install -q "openvino>=2023.3.0" "nncf>=2.7" + %pip install datasets "torchmetrics>=0.11.0" "torch>=2.1.0" --extra-index-url https://download.pytorch.org/whl/cpu + %pip install -q soundfile librosa "transformers>=4.36.2" --extra-index-url https://download.pytorch.org/whl/cpu .. code:: ipython3 from transformers import Wav2Vec2Processor, Data2VecAudioForCTC - + processor = Wav2Vec2Processor.from_pretrained("facebook/data2vec-audio-base-960h") model = Data2VecAudioForCTC.from_pretrained("facebook/data2vec-audio-base-960h") - -.. parsed-literal:: - - 2023-09-12 19:27:57.776647: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-09-12 19:27:57.812053: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. - To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-09-12 19:27:58.411557: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT - - Convert model to OpenVINO Intermediate Representation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -115,14 +106,14 @@ Convert model to OpenVINO Intermediate Representation import openvino as ov import torch - + core = ov.Core() - + BATCH_SIZE = 1 MAX_SEQ_LENGTH = 30480 - + ir_model_path = MODEL_DIR / "data2vec-audo-base.xml" - + if not ir_model_path.exists(): ov_model = ov.convert_model(model, example_input=torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float)) ov.save_model(ov_model, str(ir_model_path)) @@ -132,12 +123,6 @@ Convert model to OpenVINO Intermediate Representation ov_model = core.read_model(ir_model_path) - -.. parsed-literal:: - - Read IR model from model/data2vec-audo-base.xml - - Prepare inference data ~~~~~~~~~~~~~~~~~~~~~~ @@ -152,30 +137,23 @@ dataset. .. code:: ipython3 from datasets import load_dataset - + ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") - - + + # define preprocessing function for converting audio to input values for model def map_to_input(batch): preprocessed_signal = processor(batch["audio"]["array"], return_tensors="pt", padding="longest", sampling_rate=batch['audio']['sampling_rate']) input_values = preprocessed_signal.input_values batch['input_values'] = input_values return batch - - + + # apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore dataset = ds.map(map_to_input, batched=False, remove_columns=["audio"]) - + test_sample = ds[0]["audio"] - -.. parsed-literal:: - - Found cached dataset librispeech_asr_dummy (/home/ea/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc) - Loading cached processed dataset at /home/ea/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc/cache-5282243604a7a526.arrow - - Check model inference result ---------------------------- @@ -194,8 +172,8 @@ For reference, see the same function provided for OpenVINO model. .. code:: ipython3 import numpy as np - - + + # inference function for pytorch def torch_infer(model, sample): logits = model(torch.Tensor(sample['input_values'])).logits @@ -203,8 +181,8 @@ For reference, see the same function provided for OpenVINO model. predicted_ids = torch.argmax(logits, dim=-1) transcription = processor.batch_decode(predicted_ids) return transcription - - + + # inference function for openvino def ov_infer(model, sample): output = model.output(0) @@ -216,7 +194,7 @@ For reference, see the same function provided for OpenVINO model. .. code:: ipython3 core = ov.Core() - + pt_transcription = torch_infer(model, dataset[0]) compiled_model = core.compile_model(ov_model) ov_transcription = ov_infer(compiled_model, dataset[0]) @@ -224,7 +202,7 @@ For reference, see the same function provided for OpenVINO model. .. code:: ipython3 import IPython.display as ipd - + print(f"[Reference]: {dataset[0]['text']}") print(f"[PyTorch]: {pt_transcription[0]}") print(f"[OpenVINO FP16]: {ov_transcription[0]}") @@ -242,7 +220,7 @@ For reference, see the same function provided for OpenVINO model. .. raw:: html - +
@@ -590,16 +634,48 @@ Guide `__ - `Prepare Dataset <#prepare-dataset>`__ @@ -29,7 +29,7 @@ This tutorial consists of the following steps: - `Create Dataset for Validation <#create-dataset-for-validation>`__ - `Run nncf.quantize for Getting an Optimized - Model <#run-nncfquantize-for-getting-an-optimized-model>`__ + Model <#run-nncf-quantize-for-getting-an-optimized-model>`__ - `Serialize an OpenVINO IR model <#serialize-an-openvino-ir-model>`__ - `Compare Accuracy of the Original and Quantized Models <#compare-accuracy-of-the-original-and-quantized-models>`__ @@ -55,12 +55,12 @@ This tutorial consists of the following steps: .. code:: ipython3 from pathlib import Path - + # Set the data and model directories DATA_DIR = Path("data") MODEL_DIR = Path('model') model_repo = 'pytorch-cifar-models' - + DATA_DIR.mkdir(exist_ok=True) MODEL_DIR.mkdir(exist_ok=True) @@ -79,28 +79,406 @@ Model preparation stage has the following steps: .. code:: ipython3 import sys - + if not Path(model_repo).exists(): !git clone https://github.com/chenyaofo/pytorch-cifar-models.git - + sys.path.append(model_repo) .. parsed-literal:: Cloning into 'pytorch-cifar-models'... + + +.. parsed-literal:: + remote: Enumerating objects: 282, done. - remote: Counting objects: 100% (281/281), done. - remote: Compressing objects: 100% (96/96), done. + remote: Counting objects: 0% (1/281) +remote: Counting objects: 1% (3/281) +remote: Counting objects: 2% (6/281) +remote: Counting objects: 3% (9/281) +remote: Counting objects: 4% (12/281) +remote: Counting objects: 5% (15/281) +remote: Counting objects: 6% (17/281) +remote: Counting objects: 7% (20/281) +remote: Counting objects: 8% (23/281) +remote: Counting objects: 9% (26/281) +remote: Counting objects: 10% (29/281) +remote: Counting objects: 11% (31/281) +remote: Counting objects: 12% (34/281) +remote: Counting objects: 13% (37/281) +remote: Counting objects: 14% (40/281) +remote: Counting objects: 15% (43/281) +remote: Counting objects: 16% (45/281) +remote: Counting objects: 17% (48/281) +remote: Counting objects: 18% (51/281) +remote: Counting objects: 19% (54/281) +remote: Counting objects: 20% (57/281) +remote: Counting objects: 21% (60/281) +remote: Counting objects: 22% (62/281) +remote: Counting objects: 23% (65/281) +remote: Counting objects: 24% (68/281) +remote: Counting objects: 25% (71/281) +remote: Counting objects: 26% (74/281) +remote: Counting objects: 27% (76/281) +remote: Counting objects: 28% (79/281) +remote: Counting objects: 29% (82/281) +remote: Counting objects: 30% (85/281) +remote: Counting objects: 31% (88/281) +remote: Counting objects: 32% (90/281) +remote: Counting objects: 33% (93/281) +remote: Counting objects: 34% (96/281) +remote: Counting objects: 35% (99/281) +remote: Counting objects: 36% (102/281) +remote: Counting objects: 37% (104/281) +remote: Counting objects: 38% (107/281) +remote: Counting objects: 39% (110/281) +remote: Counting objects: 40% (113/281) +remote: Counting objects: 41% (116/281) +remote: Counting objects: 42% (119/281) +remote: Counting objects: 43% (121/281) +remote: Counting objects: 44% (124/281) +remote: Counting objects: 45% (127/281) +remote: Counting objects: 46% (130/281) +remote: Counting objects: 47% (133/281) +remote: Counting objects: 48% (135/281) +remote: Counting objects: 49% (138/281) +remote: Counting objects: 50% (141/281) +remote: Counting objects: 51% (144/281) +remote: Counting objects: 52% (147/281) +remote: Counting objects: 53% (149/281) +remote: Counting objects: 54% (152/281) +remote: Counting objects: 55% (155/281) +remote: Counting objects: 56% (158/281) +remote: Counting objects: 57% (161/281) +remote: Counting objects: 58% (163/281) +remote: Counting objects: 59% (166/281) +remote: Counting objects: 60% (169/281) +remote: Counting objects: 61% (172/281) +remote: Counting objects: 62% (175/281) +remote: Counting objects: 63% (178/281) +remote: Counting objects: 64% (180/281) +remote: Counting objects: 65% (183/281) +remote: Counting objects: 66% (186/281) +remote: Counting objects: 67% (189/281) +remote: Counting objects: 68% (192/281) +remote: Counting objects: 69% (194/281) +remote: Counting objects: 70% (197/281) +remote: Counting objects: 71% (200/281) +remote: Counting objects: 72% (203/281) +remote: Counting objects: 73% (206/281) +remote: Counting objects: 74% (208/281) +remote: Counting objects: 75% (211/281) +remote: Counting objects: 76% (214/281) +remote: Counting objects: 77% (217/281) +remote: Counting objects: 78% (220/281) +remote: Counting objects: 79% (222/281) +remote: Counting objects: 80% (225/281) +remote: Counting objects: 81% (228/281) +remote: Counting objects: 82% (231/281) +remote: Counting objects: 83% (234/281) +remote: Counting objects: 84% (237/281) +remote: Counting objects: 85% (239/281) +remote: Counting objects: 86% (242/281) +remote: Counting objects: 87% (245/281) +remote: Counting objects: 88% (248/281) +remote: Counting objects: 89% (251/281) +remote: Counting objects: 90% (253/281) +remote: Counting objects: 91% (256/281) +remote: Counting objects: 92% (259/281) +remote: Counting objects: 93% (262/281) +remote: Counting objects: 94% (265/281) +remote: Counting objects: 95% (267/281) +remote: Counting objects: 96% (270/281) +remote: Counting objects: 97% (273/281) +remote: Counting objects: 98% (276/281) +remote: Counting objects: 99% (279/281) +remote: Counting objects: 100% (281/281) +remote: Counting objects: 100% (281/281), done. + remote: Compressing objects: 1% (1/96) +remote: Compressing objects: 2% (2/96) +remote: Compressing objects: 3% (3/96) +remote: Compressing objects: 4% (4/96) +remote: Compressing objects: 5% (5/96) +remote: Compressing objects: 6% (6/96) +remote: Compressing objects: 7% (7/96) +remote: Compressing objects: 8% (8/96) +remote: Compressing objects: 9% (9/96) +remote: Compressing objects: 10% (10/96) +remote: Compressing objects: 11% (11/96) +remote: Compressing objects: 12% (12/96) +remote: Compressing objects: 13% (13/96) +remote: Compressing objects: 14% (14/96) +remote: Compressing objects: 15% (15/96) +remote: Compressing objects: 16% (16/96) +remote: Compressing objects: 17% (17/96) +remote: Compressing objects: 18% (18/96) +remote: Compressing objects: 19% (19/96) +remote: Compressing objects: 20% (20/96) +remote: Compressing objects: 21% (21/96) +remote: Compressing objects: 22% (22/96) +remote: Compressing objects: 23% (23/96) +remote: Compressing objects: 25% (24/96) +remote: Compressing objects: 26% (25/96) +remote: Compressing objects: 27% (26/96) +remote: Compressing objects: 28% (27/96) +remote: Compressing objects: 29% (28/96) +remote: Compressing objects: 30% (29/96) +remote: Compressing objects: 31% (30/96) +remote: Compressing objects: 32% (31/96) +remote: Compressing objects: 33% (32/96) +remote: Compressing objects: 34% (33/96) +remote: Compressing objects: 35% (34/96) +remote: Compressing objects: 36% (35/96) +remote: Compressing objects: 37% (36/96) +remote: Compressing objects: 38% (37/96) +remote: Compressing objects: 39% (38/96) +remote: Compressing objects: 40% (39/96) +remote: Compressing objects: 41% (40/96) +remote: Compressing objects: 42% (41/96) +remote: Compressing objects: 43% (42/96) +remote: Compressing objects: 44% (43/96) +remote: Compressing objects: 45% (44/96) +remote: Compressing objects: 46% (45/96) +remote: Compressing objects: 47% (46/96) +remote: Compressing objects: 48% (47/96) +remote: Compressing objects: 50% (48/96) +remote: Compressing objects: 51% (49/96) +remote: Compressing objects: 52% (50/96) +remote: Compressing objects: 53% (51/96) +remote: Compressing objects: 54% (52/96) +remote: Compressing objects: 55% (53/96) +remote: Compressing objects: 56% (54/96) +remote: Compressing objects: 57% (55/96) +remote: Compressing objects: 58% (56/96) +remote: Compressing objects: 59% (57/96) +remote: Compressing objects: 60% (58/96) +remote: Compressing objects: 61% (59/96) +remote: Compressing objects: 62% (60/96) +remote: Compressing objects: 63% (61/96) +remote: Compressing objects: 64% (62/96) +remote: Compressing objects: 65% (63/96) +remote: Compressing objects: 66% (64/96) +remote: Compressing objects: 67% (65/96) +remote: Compressing objects: 68% (66/96) +remote: Compressing objects: 69% (67/96) +remote: Compressing objects: 70% (68/96) +remote: Compressing objects: 71% (69/96) +remote: Compressing objects: 72% (70/96) +remote: Compressing objects: 73% (71/96) +remote: Compressing objects: 75% (72/96) +remote: Compressing objects: 76% (73/96) +remote: Compressing objects: 77% (74/96) +remote: Compressing objects: 78% (75/96) +remote: Compressing objects: 79% (76/96) +remote: Compressing objects: 80% (77/96) +remote: Compressing objects: 81% (78/96) +remote: Compressing objects: 82% (79/96) +remote: Compressing objects: 83% (80/96) +remote: Compressing objects: 84% (81/96) +remote: Compressing objects: 85% (82/96) +remote: Compressing objects: 86% (83/96) +remote: Compressing objects: 87% (84/96) +remote: Compressing objects: 88% (85/96) +remote: Compressing objects: 89% (86/96) +remote: Compressing objects: 90% (87/96) +remote: Compressing objects: 91% (88/96) +remote: Compressing objects: 92% (89/96) +remote: Compressing objects: 93% (90/96) +remote: Compressing objects: 94% (91/96) +remote: Compressing objects: 95% (92/96) +remote: Compressing objects: 96% (93/96) +remote: Compressing objects: 97% (94/96) +remote: Compressing objects: 98% (95/96) +remote: Compressing objects: 100% (96/96) +remote: Compressing objects: 100% (96/96), done. + Receiving objects: 0% (1/282) +Receiving objects: 1% (3/282) +Receiving objects: 2% (6/282) +Receiving objects: 3% (9/282) +Receiving objects: 4% (12/282) +Receiving objects: 5% (15/282) +Receiving objects: 6% (17/282) +Receiving objects: 7% (20/282) +Receiving objects: 8% (23/282) +Receiving objects: 9% (26/282) +Receiving objects: 10% (29/282) +Receiving objects: 11% (32/282) +Receiving objects: 12% (34/282) +Receiving objects: 13% (37/282) +Receiving objects: 14% (40/282) +Receiving objects: 15% (43/282) +Receiving objects: 16% (46/282) +Receiving objects: 17% (48/282) +Receiving objects: 18% (51/282) +Receiving objects: 19% (54/282) +Receiving objects: 20% (57/282) +Receiving objects: 21% (60/282) +Receiving objects: 22% (63/282) +Receiving objects: 23% (65/282) +Receiving objects: 24% (68/282) +Receiving objects: 25% (71/282) +Receiving objects: 26% (74/282) +Receiving objects: 27% (77/282) +Receiving objects: 28% (79/282) +Receiving objects: 29% (82/282) +Receiving objects: 30% (85/282) +Receiving objects: 31% (88/282) +Receiving objects: 32% (91/282) +Receiving objects: 33% (94/282) +Receiving objects: 34% (96/282) +Receiving objects: 35% (99/282) +Receiving objects: 36% (102/282) +Receiving objects: 37% (105/282) +Receiving objects: 38% (108/282) +Receiving objects: 39% (110/282) +Receiving objects: 40% (113/282) +Receiving objects: 41% (116/282) +Receiving objects: 42% (119/282) +Receiving objects: 43% (122/282) +Receiving objects: 44% (125/282) +Receiving objects: 45% (127/282) +Receiving objects: 46% (130/282) +Receiving objects: 47% (133/282) +Receiving objects: 48% (136/282) +Receiving objects: 49% (139/282) +Receiving objects: 50% (141/282) +Receiving objects: 51% (144/282) +Receiving objects: 52% (147/282) +Receiving objects: 53% (150/282) +Receiving objects: 54% (153/282) +Receiving objects: 55% (156/282) +Receiving objects: 56% (158/282) +Receiving objects: 57% (161/282) +Receiving objects: 58% (164/282) +Receiving objects: 59% (167/282) +Receiving objects: 60% (170/282) +Receiving objects: 61% (173/282) +Receiving objects: 62% (175/282) +Receiving objects: 63% (178/282) +Receiving objects: 64% (181/282) +Receiving objects: 65% (184/282) +Receiving objects: 66% (187/282) +Receiving objects: 67% (189/282) +Receiving objects: 68% (192/282) +Receiving objects: 69% (195/282) +Receiving objects: 70% (198/282) +Receiving objects: 71% (201/282) +Receiving objects: 72% (204/282) +Receiving objects: 73% (206/282) +Receiving objects: 74% (209/282) +Receiving objects: 75% (212/282) + +.. parsed-literal:: + + Receiving objects: 76% (215/282) + +.. parsed-literal:: + + Receiving objects: 77% (218/282) + +.. parsed-literal:: + + Receiving objects: 78% (220/282), 1.64 MiB | 3.23 MiB/s + +.. parsed-literal:: + + Receiving objects: 79% (223/282), 1.64 MiB | 3.23 MiB/s + +.. parsed-literal:: + + Receiving objects: 80% (226/282), 1.64 MiB | 3.23 MiB/s + +.. parsed-literal:: + + Receiving objects: 80% (227/282), 3.47 MiB | 3.40 MiB/s + +.. parsed-literal:: + + Receiving objects: 81% (229/282), 3.47 MiB | 3.40 MiB/s + +.. parsed-literal:: + + Receiving objects: 82% (232/282), 3.47 MiB | 3.40 MiB/s + +.. parsed-literal:: + + Receiving objects: 83% (235/282), 3.47 MiB | 3.40 MiB/s + +.. parsed-literal:: + + Receiving objects: 84% (237/282), 5.31 MiB | 3.45 MiB/s + +.. parsed-literal:: + + Receiving objects: 85% (240/282), 5.31 MiB | 3.45 MiB/s + +.. parsed-literal:: + + Receiving objects: 86% (243/282), 5.31 MiB | 3.45 MiB/s + +.. parsed-literal:: + + Receiving objects: 87% (246/282), 7.14 MiB | 3.47 MiB/s + +.. parsed-literal:: + + Receiving objects: 88% (249/282), 7.14 MiB | 3.47 MiB/s + +.. parsed-literal:: + + Receiving objects: 89% (251/282), 7.14 MiB | 3.47 MiB/s + +.. parsed-literal:: + remote: Total 282 (delta 135), reused 269 (delta 128), pack-reused 1 - Receiving objects: 100% (282/282), 9.22 MiB | 3.72 MiB/s, done. - Resolving deltas: 100% (135/135), done. + Receiving objects: 90% (254/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 91% (257/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 92% (260/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 93% (263/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 94% (266/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 95% (268/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 96% (271/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 97% (274/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 98% (277/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 99% (280/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 100% (282/282), 8.97 MiB | 3.49 MiB/s +Receiving objects: 100% (282/282), 9.22 MiB | 3.52 MiB/s, done. + Resolving deltas: 0% (0/135) +Resolving deltas: 2% (4/135) +Resolving deltas: 5% (7/135) +Resolving deltas: 6% (9/135) +Resolving deltas: 11% (16/135) +Resolving deltas: 17% (24/135) +Resolving deltas: 18% (25/135) +Resolving deltas: 23% (32/135) +Resolving deltas: 25% (34/135) +Resolving deltas: 27% (37/135) +Resolving deltas: 28% (38/135) +Resolving deltas: 34% (46/135) +Resolving deltas: 40% (54/135) +Resolving deltas: 45% (62/135) +Resolving deltas: 46% (63/135) +Resolving deltas: 50% (68/135) +Resolving deltas: 57% (77/135) +Resolving deltas: 58% (79/135) +Resolving deltas: 59% (80/135) +Resolving deltas: 60% (81/135) +Resolving deltas: 61% (83/135) +Resolving deltas: 66% (90/135) +Resolving deltas: 71% (97/135) + +.. parsed-literal:: + + Resolving deltas: 100% (135/135) +Resolving deltas: 100% (135/135), done. .. code:: ipython3 from pytorch_cifar_models import cifar10_mobilenetv2_x1_0 - + model = cifar10_mobilenetv2_x1_0(pretrained=True) OpenVINO supports PyTorch models via conversion to OpenVINO Intermediate @@ -118,12 +496,12 @@ can be found on this .. code:: ipython3 import openvino as ov - + model.eval() - + ov_model = ov.convert_model(model, input=[1,3,32,32]) - - ov.save_model(ov_model, MODEL_DIR / "mobilenet_v2.xml") + + ov.save_model(ov_model, MODEL_DIR / "mobilenet_v2.xml") Prepare Dataset --------------- @@ -141,7 +519,7 @@ Preprocessing for model obtained from training import torch from torchvision import transforms from torchvision.datasets import CIFAR10 - + transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.4914, 0.4822, 0.4465), (0.247, 0.243, 0.261))]) dataset = CIFAR10(root=DATA_DIR, train=False, transform=transform, download=True) val_loader = torch.utils.data.DataLoader( @@ -160,7 +538,2209 @@ Preprocessing for model obtained from training .. parsed-literal:: - 100%|██████████| 170498071/170498071 [00:48<00:00, 3527631.36it/s] + + 0%| | 0/170498071 [00:00 -
@@ -271,10 +2852,7 @@ about supported parameters can be found on this -.. raw:: html -
-    
@@ -300,7 +2878,7 @@ Compare Accuracy of the Original and Quantized Models from tqdm.notebook import tqdm import numpy as np - + def test_accuracy(ov_model, data_loader): correct = 0 total = 0 @@ -321,7 +2899,7 @@ select device from dropdown list for running inference using OpenVINO .. code:: ipython3 import ipywidgets as widgets - + core = ov.Core() device = widgets.Dropdown( options=core.available_devices + ["AUTO"], @@ -329,7 +2907,7 @@ select device from dropdown list for running inference using OpenVINO description='Device:', disabled=False, ) - + device @@ -346,7 +2924,7 @@ select device from dropdown list for running inference using OpenVINO core = ov.Core() compiled_model = core.compile_model(ov_model, device.value) optimized_compiled_model = core.compile_model(quant_ov_model, device.value) - + orig_accuracy = test_accuracy(compiled_model, val_loader) optimized_accuracy = test_accuracy(optimized_compiled_model, val_loader) @@ -404,18 +2982,18 @@ Tool `__ [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: [ INFO ] AUTO - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 9.65 ms + [ INFO ] Read model took 9.59 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] x (node: x) : f32 / [...] / [1,3,32,32] @@ -429,7 +3007,11 @@ Tool `__ [ INFO ] Model outputs: [ INFO ] x.17 (node: aten::linear/Add) : f32 / [...] / [1,10] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 176.84 ms + + +.. parsed-literal:: + + [ INFO ] Compile model took 215.86 ms [Step 8/11] Querying optimal runtime parameters [ INFO ] Model: [ INFO ] NETWORK_NAME: Model2 @@ -450,28 +3032,36 @@ Tool `__ [ INFO ] NETWORK_NAME: Model2 [ INFO ] NUM_STREAMS: 12 [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12 - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT + [ INFO ] PERFORMANCE_HINT: THROUGHPUT [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 - [ INFO ] PERF_COUNT: False + [ INFO ] PERF_COUNT: NO [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE [ INFO ] MODEL_PRIORITY: Priority.MEDIUM [ INFO ] LOADED_FROM_CACHE: False [Step 9/11] Creating infer requests and preparing input tensors [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values! - [ INFO ] Fill input 'x' with random values + [ INFO ] Fill input 'x' with random values [Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration) [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 3.23 ms + + +.. parsed-literal:: + + [ INFO ] First inference took 3.33 ms + + +.. parsed-literal:: + [Step 11/11] Dumping statistics report [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 87924 iterations - [ INFO ] Duration: 15003.28 ms + [ INFO ] Count: 88704 iterations + [ INFO ] Duration: 15002.65 ms [ INFO ] Latency: - [ INFO ] Median: 1.82 ms - [ INFO ] Average: 1.84 ms - [ INFO ] Min: 1.20 ms + [ INFO ] Median: 1.84 ms + [ INFO ] Average: 1.85 ms + [ INFO ] Min: 1.19 ms [ INFO ] Max: 8.90 ms - [ INFO ] Throughput: 5860.32 FPS + [ INFO ] Throughput: 5912.55 FPS .. code:: ipython3 @@ -486,18 +3076,18 @@ Tool `__ [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: [ INFO ] AUTO - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 18.89 ms + [ INFO ] Read model took 19.08 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] x (node: x) : f32 / [...] / [1,3,32,32] @@ -511,7 +3101,11 @@ Tool `__ [ INFO ] Model outputs: [ INFO ] x.17 (node: aten::linear/Add) : f32 / [...] / [1,10] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 344.83 ms + + +.. parsed-literal:: + + [ INFO ] Compile model took 331.25 ms [Step 8/11] Querying optimal runtime parameters [ INFO ] Model: [ INFO ] NETWORK_NAME: Model2 @@ -532,28 +3126,32 @@ Tool `__ [ INFO ] NETWORK_NAME: Model2 [ INFO ] NUM_STREAMS: 12 [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12 - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT + [ INFO ] PERFORMANCE_HINT: THROUGHPUT [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 - [ INFO ] PERF_COUNT: False + [ INFO ] PERF_COUNT: NO [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE [ INFO ] MODEL_PRIORITY: Priority.MEDIUM [ INFO ] LOADED_FROM_CACHE: False [Step 9/11] Creating infer requests and preparing input tensors [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values! - [ INFO ] Fill input 'x' with random values + [ INFO ] Fill input 'x' with random values [Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration) [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 2.03 ms + [ INFO ] First inference took 1.87 ms + + +.. parsed-literal:: + [Step 11/11] Dumping statistics report [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 163332 iterations - [ INFO ] Duration: 15000.99 ms + [ INFO ] Count: 167340 iterations + [ INFO ] Duration: 15001.30 ms [ INFO ] Latency: - [ INFO ] Median: 1.02 ms - [ INFO ] Average: 1.06 ms - [ INFO ] Min: 0.71 ms - [ INFO ] Max: 15.01 ms - [ INFO ] Throughput: 10888.08 FPS + [ INFO ] Median: 1.00 ms + [ INFO ] Average: 1.03 ms + [ INFO ] Min: 0.72 ms + [ INFO ] Max: 7.10 ms + [ INFO ] Throughput: 11155.04 FPS Compare results on four pictures @@ -567,7 +3165,7 @@ Compare results on four pictures labels_names = ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"] all_pictures = [] all_labels = [] - + # Get all pictures and their labels. for i, batch in enumerate(val_loader): all_pictures.append(batch[0].numpy()) @@ -576,7 +3174,7 @@ Compare results on four pictures .. code:: ipython3 import matplotlib.pyplot as plt - + def plot_pictures(indexes: list, all_pictures=all_pictures, all_labels=all_labels): """Plot 4 pictures. :param indexes: a list of indexes of pictures to be displayed. @@ -589,19 +3187,19 @@ Compare results on four pictures assert idx < 10000, 'Cannot get such index, there are only 10000' pic = np.rollaxis(all_pictures[idx].squeeze(), 0, 3) images.append(pic) - + labels.append(labels_names[all_labels[idx]]) - + f, axarr = plt.subplots(1, 4) axarr[0].imshow(images[0]) axarr[0].set_title(labels[0]) - + axarr[1].imshow(images[1]) axarr[1].set_title(labels[1]) - + axarr[2].imshow(images[2]) axarr[2].set_title(labels[2]) - + axarr[3].imshow(images[3]) axarr[3].set_title(labels[3]) @@ -610,7 +3208,7 @@ Compare results on four pictures def infer_on_pictures(model, indexes: list, all_pictures=all_pictures): """ Inference model on a few pictures. :param net: model on which do inference - :param indexes: list of indexes + :param indexes: list of indexes """ output_key = model.output(0) predicted_labels = [] @@ -624,12 +3222,12 @@ Compare results on four pictures .. code:: ipython3 indexes_to_infer = [7, 12, 15, 20] # To plot, specify 4 indexes. - + plot_pictures(indexes_to_infer) - + results_float = infer_on_pictures(compiled_model, indexes_to_infer) results_quanized = infer_on_pictures(optimized_compiled_model, indexes_to_infer) - + print(f"Labels for picture from float model : {results_float}.") print(f"Labels for picture from quantized model : {results_quanized}.") @@ -637,8 +3235,20 @@ Compare results on four pictures .. parsed-literal:: Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). + + +.. parsed-literal:: + Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). + + +.. parsed-literal:: + Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). + + +.. parsed-literal:: + Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). @@ -649,5 +3259,5 @@ Compare results on four pictures -.. image:: 113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_2.png +.. image:: 113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_5.png diff --git a/docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_2.png b/docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_5.png similarity index 100% rename from docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_2.png rename to docs/notebooks/113-image-classification-quantization-with-output_files/113-image-classification-quantization-with-output_30_5.png diff --git a/docs/notebooks/113-image-classification-quantization-with-output_files/index.html b/docs/notebooks/113-image-classification-quantization-with-output_files/index.html deleted file mode 100644 index f6301652c6b..00000000000 --- a/docs/notebooks/113-image-classification-quantization-with-output_files/index.html +++ /dev/null @@ -1,7 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/113-image-classification-quantization-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/113-image-classification-quantization-with-output_files/


../
-113-image-classification-quantization-with-outp..> 07-Dec-2023 00:49               14855
-

- diff --git a/docs/notebooks/113-image-classification-quantization-with-output_files/index.html.tmp b/docs/notebooks/113-image-classification-quantization-with-output_files/index.html.tmp new file mode 100644 index 00000000000..9fbe70d140f --- /dev/null +++ b/docs/notebooks/113-image-classification-quantization-with-output_files/index.html.tmp @@ -0,0 +1,7 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/113-image-classification-quantization-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/113-image-classification-quantization-with-output_files/


../
+113-image-classification-quantization-with-outp..> 26-Jan-2024 01:05               14855
+

+ diff --git a/docs/notebooks/115-async-api-with-output.rst b/docs/notebooks/115-async-api-with-output.rst index a3378d228e5..578b1afafdd 100644 --- a/docs/notebooks/115-async-api-with-output.rst +++ b/docs/notebooks/115-async-api-with-output.rst @@ -11,8 +11,8 @@ device is busy with inference, the application can perform other tasks in parallel (for example, populating inputs or scheduling other requests) rather than wait for the current inference to complete first. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Imports <#imports>`__ - `Prepare model and data @@ -37,7 +37,8 @@ requests) rather than wait for the current inference to complete first. - `AsyncInferQueue <#asyncinferqueue>`__ - `Setting Callback <#setting-callback>`__ - - `Test the performance with AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__ + - `Test the performance with + AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__ Imports ------- @@ -53,6 +54,10 @@ Imports .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. @@ -113,10 +118,204 @@ each frame of the video. ################|| Downloading person-detection-0202 ||################ ========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.xml - + + +.. parsed-literal:: + + ... 12%, 32 KB, 1452 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 25%, 64 KB, 1385 KB/s, 0 seconds passed +... 38%, 96 KB, 1632 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 51%, 128 KB, 1424 KB/s, 0 seconds passed +... 64%, 160 KB, 1393 KB/s, 0 seconds passed +... 77%, 192 KB, 1664 KB/s, 0 seconds passed +... 89%, 224 KB, 1795 KB/s, 0 seconds passed +... 100%, 248 KB, 1986 KB/s, 0 seconds passed + + + +.. parsed-literal:: + ========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.bin - + + +.. parsed-literal:: + + ... 0%, 32 KB, 1284 KB/s, 0 seconds passed +... 1%, 64 KB, 1284 KB/s, 0 seconds passed +... 2%, 96 KB, 1894 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 3%, 128 KB, 1699 KB/s, 0 seconds passed +... 4%, 160 KB, 2092 KB/s, 0 seconds passed +... 5%, 192 KB, 2274 KB/s, 0 seconds passed +... 6%, 224 KB, 2623 KB/s, 0 seconds passed +... 7%, 256 KB, 2328 KB/s, 0 seconds passed +... 8%, 288 KB, 2599 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 9%, 320 KB, 2693 KB/s, 0 seconds passed +... 9%, 352 KB, 2939 KB/s, 0 seconds passed +... 10%, 384 KB, 2655 KB/s, 0 seconds passed +... 11%, 416 KB, 2858 KB/s, 0 seconds passed +... 12%, 448 KB, 2918 KB/s, 0 seconds passed +... 13%, 480 KB, 3100 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 14%, 512 KB, 2858 KB/s, 0 seconds passed +... 15%, 544 KB, 2904 KB/s, 0 seconds passed +... 16%, 576 KB, 3065 KB/s, 0 seconds passed +... 17%, 608 KB, 3219 KB/s, 0 seconds passed +... 18%, 640 KB, 2998 KB/s, 0 seconds passed +... 18%, 672 KB, 3133 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 19%, 704 KB, 3164 KB/s, 0 seconds passed +... 20%, 736 KB, 3293 KB/s, 0 seconds passed +... 21%, 768 KB, 3098 KB/s, 0 seconds passed +... 22%, 800 KB, 3120 KB/s, 0 seconds passed +... 23%, 832 KB, 3228 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 24%, 864 KB, 3062 KB/s, 0 seconds passed +... 25%, 896 KB, 3170 KB/s, 0 seconds passed +... 26%, 928 KB, 3274 KB/s, 0 seconds passed +... 27%, 960 KB, 3291 KB/s, 0 seconds passed +... 27%, 992 KB, 3388 KB/s, 0 seconds passed +... 28%, 1024 KB, 3231 KB/s, 0 seconds passed +... 29%, 1056 KB, 3322 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 30%, 1088 KB, 3331 KB/s, 0 seconds passed +... 31%, 1120 KB, 3189 KB/s, 0 seconds passed +... 32%, 1152 KB, 3275 KB/s, 0 seconds passed +... 33%, 1184 KB, 3360 KB/s, 0 seconds passed +... 34%, 1216 KB, 3370 KB/s, 0 seconds passed +... 35%, 1248 KB, 3448 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 36%, 1280 KB, 3316 KB/s, 0 seconds passed +... 36%, 1312 KB, 3390 KB/s, 0 seconds passed +... 37%, 1344 KB, 3401 KB/s, 0 seconds passed +... 38%, 1376 KB, 3472 KB/s, 0 seconds passed +... 39%, 1408 KB, 3349 KB/s, 0 seconds passed +... 40%, 1440 KB, 3418 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 41%, 1472 KB, 3426 KB/s, 0 seconds passed +... 42%, 1504 KB, 3488 KB/s, 0 seconds passed +... 43%, 1536 KB, 3372 KB/s, 0 seconds passed +... 44%, 1568 KB, 3437 KB/s, 0 seconds passed +... 45%, 1600 KB, 3446 KB/s, 0 seconds passed +... 45%, 1632 KB, 3505 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 46%, 1664 KB, 3400 KB/s, 0 seconds passed +... 47%, 1696 KB, 3457 KB/s, 0 seconds passed +... 48%, 1728 KB, 3468 KB/s, 0 seconds passed +... 49%, 1760 KB, 3520 KB/s, 0 seconds passed +... 50%, 1792 KB, 3417 KB/s, 0 seconds passed +... 51%, 1824 KB, 3472 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 52%, 1856 KB, 3478 KB/s, 0 seconds passed +... 53%, 1888 KB, 3383 KB/s, 0 seconds passed +... 54%, 1920 KB, 3437 KB/s, 0 seconds passed +... 54%, 1952 KB, 3487 KB/s, 0 seconds passed +... 55%, 1984 KB, 3495 KB/s, 0 seconds passed +... 56%, 2016 KB, 3544 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 57%, 2048 KB, 3451 KB/s, 0 seconds passed +... 58%, 2080 KB, 3500 KB/s, 0 seconds passed +... 59%, 2112 KB, 3506 KB/s, 0 seconds passed +... 60%, 2144 KB, 3554 KB/s, 0 seconds passed +... 61%, 2176 KB, 3466 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 62%, 2208 KB, 3510 KB/s, 0 seconds passed +... 63%, 2240 KB, 3513 KB/s, 0 seconds passed +... 64%, 2272 KB, 3561 KB/s, 0 seconds passed +... 64%, 2304 KB, 3478 KB/s, 0 seconds passed +... 65%, 2336 KB, 3518 KB/s, 0 seconds passed +... 66%, 2368 KB, 3521 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 67%, 2400 KB, 3444 KB/s, 0 seconds passed +... 68%, 2432 KB, 3486 KB/s, 0 seconds passed +... 69%, 2464 KB, 3495 KB/s, 0 seconds passed +... 70%, 2496 KB, 3528 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 71%, 2528 KB, 3454 KB/s, 0 seconds passed +... 72%, 2560 KB, 3496 KB/s, 0 seconds passed +... 73%, 2592 KB, 3505 KB/s, 0 seconds passed +... 73%, 2624 KB, 3515 KB/s, 0 seconds passed +... 74%, 2656 KB, 3463 KB/s, 0 seconds passed +... 75%, 2688 KB, 3502 KB/s, 0 seconds passed +... 76%, 2720 KB, 3514 KB/s, 0 seconds passed +... 77%, 2752 KB, 3523 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 78%, 2784 KB, 3474 KB/s, 0 seconds passed +... 79%, 2816 KB, 3510 KB/s, 0 seconds passed +... 80%, 2848 KB, 3522 KB/s, 0 seconds passed +... 81%, 2880 KB, 3530 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 82%, 2912 KB, 3482 KB/s, 0 seconds passed +... 82%, 2944 KB, 3518 KB/s, 0 seconds passed +... 83%, 2976 KB, 3529 KB/s, 0 seconds passed +... 84%, 3008 KB, 3535 KB/s, 0 seconds passed +... 85%, 3040 KB, 3489 KB/s, 0 seconds passed +... 86%, 3072 KB, 3523 KB/s, 0 seconds passed +... 87%, 3104 KB, 3532 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 88%, 3136 KB, 3540 KB/s, 0 seconds passed +... 89%, 3168 KB, 3498 KB/s, 0 seconds passed +... 90%, 3200 KB, 3530 KB/s, 0 seconds passed +... 91%, 3232 KB, 3539 KB/s, 0 seconds passed +... 91%, 3264 KB, 3544 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 92%, 3296 KB, 3504 KB/s, 0 seconds passed +... 93%, 3328 KB, 3536 KB/s, 0 seconds passed +... 94%, 3360 KB, 3544 KB/s, 0 seconds passed +... 95%, 3392 KB, 3550 KB/s, 0 seconds passed +... 96%, 3424 KB, 3509 KB/s, 0 seconds passed +... 97%, 3456 KB, 3540 KB/s, 0 seconds passed +... 98%, 3488 KB, 3548 KB/s, 0 seconds passed + +.. parsed-literal:: + + ... 99%, 3520 KB, 3560 KB/s, 0 seconds passed +... 100%, 3549 KB, 3521 KB/s, 1 seconds passed + @@ -313,7 +512,7 @@ Test performance in Sync Mode .. parsed-literal:: Source ended - average throuput in sync mode: 40.67 fps + average throuput in sync mode: 43.51 fps Async Mode @@ -452,7 +651,7 @@ Test the performance in Async Mode .. parsed-literal:: Source ended - average throuput in async mode: 74.75 fps + average throuput in async mode: 74.01 fps Compare the performance @@ -595,5 +794,5 @@ Test the performance with ``AsyncInferQueue`` .. parsed-literal:: - average throughput in async mode with async infer queue: 111.75 fps + average throughput in async mode with async infer queue: 110.03 fps diff --git a/docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png b/docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png index c452c1a5664..20c300e28e4 100644 --- a/docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png +++ b/docs/notebooks/115-async-api-with-output_files/115-async-api-with-output_21_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b8c8b61f0bbb25c280a3e72cf2172fd29bf11668231cb3d2527c1b8a05307f2 -size 30406 +oid sha256:a6ef5964af038dfaf0c794dfc9a36258293ea7a6f8318f5f84490e8953a10479 +size 30402 diff --git a/docs/notebooks/115-async-api-with-output_files/index.html b/docs/notebooks/115-async-api-with-output_files/index.html deleted file mode 100644 index 376097c1329..00000000000 --- a/docs/notebooks/115-async-api-with-output_files/index.html +++ /dev/null @@ -1,10 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/115-async-api-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/115-async-api-with-output_files/


../
-115-async-api-with-output_15_0.png                 07-Dec-2023 00:49                4307
-115-async-api-with-output_19_0.png                 07-Dec-2023 00:49                4307
-115-async-api-with-output_21_0.png                 07-Dec-2023 00:49               30406
-115-async-api-with-output_27_0.png                 07-Dec-2023 00:49                4307
-

- diff --git a/docs/notebooks/115-async-api-with-output_files/index.html.tmp b/docs/notebooks/115-async-api-with-output_files/index.html.tmp new file mode 100644 index 00000000000..c63e3eb0592 --- /dev/null +++ b/docs/notebooks/115-async-api-with-output_files/index.html.tmp @@ -0,0 +1,10 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/115-async-api-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/115-async-api-with-output_files/


../
+115-async-api-with-output_15_0.png                 26-Jan-2024 01:05                4307
+115-async-api-with-output_19_0.png                 26-Jan-2024 01:05                4307
+115-async-api-with-output_21_0.png                 26-Jan-2024 01:05               30402
+115-async-api-with-output_27_0.png                 26-Jan-2024 01:05                4307
+

+ diff --git a/docs/notebooks/116-sparsity-optimization-with-output.rst b/docs/notebooks/116-sparsity-optimization-with-output.rst index 2a8b1fbc4d0..d2fbad4c9a9 100644 --- a/docs/notebooks/116-sparsity-optimization-with-output.rst +++ b/docs/notebooks/116-sparsity-optimization-with-output.rst @@ -12,7 +12,7 @@ datasets `__ using `Optimum-Intel `__. It demonstrates the inference performance advantage on 4th Gen Intel® Xeon® Scalable Processors by running it with `Sparse Weight -Decompression `__, +Decompression `__, a runtime option that seizes model sparsity for efficiency. The notebook consists of the following steps: @@ -21,8 +21,8 @@ consists of the following steps: integration with Hugging Face Optimum. - Compare sparse 8-bit vs. dense 8-bit inference performance. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Prerequisites <#prerequisites>`__ - `Imports <#imports>`__ @@ -51,6 +51,10 @@ Prerequisites .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. @@ -77,10 +81,18 @@ Imports .. parsed-literal:: No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' - 2023-12-06 23:02:39.282111: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-12-06 23:02:39.316382: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + + +.. parsed-literal:: + + 2024-01-25 23:06:01.802535: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 23:06:01.837209: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-12-06 23:02:40.030243: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + 2024-01-25 23:06:02.398511: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Download, quantize and sparsify the model, using Hugging Face Optimum API @@ -117,7 +129,11 @@ model card on Hugging Face. .. parsed-literal:: Compiling the model to CPU ... - Setting OpenVINO CACHE_DIR to /opt/home/k8sworker/.cache/huggingface/hub/models--OpenVINO--bert-base-uncased-sst2-int8-unstructured80/snapshots/dc44eb46300882463d50ee847e0f6485bad3cdad/model_cache + + +.. parsed-literal:: + + device must be of type but got instead .. parsed-literal:: @@ -193,18 +209,26 @@ as an example. It is recommended to tune based on your applications. [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 [ INFO ] [ INFO ] Device info: + + +.. parsed-literal:: + [ INFO ] CPU - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 60.26 ms + + +.. parsed-literal:: + + [ INFO ] Read model took 60.22 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?] @@ -215,7 +239,7 @@ as an example. It is recommended to tune based on your applications. [Step 5/11] Resizing model to match image sizes and given batch [ INFO ] Model batch size: 1 [ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64] - [ INFO ] Reshape model took 24.75 ms + [ INFO ] Reshape model took 23.06 ms [Step 6/11] Configuring input of the model [ INFO ] Model inputs: [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64] @@ -224,7 +248,11 @@ as an example. It is recommended to tune based on your applications. [ INFO ] Model outputs: [ INFO ] logits (node: logits) : f32 / [...] / [1,2] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 1092.05 ms + + +.. parsed-literal:: + + [ INFO ] Compile model took 1064.23 ms [Step 8/11] Querying optimal runtime parameters [ INFO ] Model: [ INFO ] NETWORK_NAME: torch_jit @@ -232,9 +260,9 @@ as an example. It is recommended to tune based on your applications. [ INFO ] NUM_STREAMS: 4 [ INFO ] AFFINITY: Affinity.CORE [ INFO ] INFERENCE_NUM_THREADS: 4 - [ INFO ] PERF_COUNT: False + [ INFO ] PERF_COUNT: NO [ INFO ] INFERENCE_PRECISION_HINT: - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT + [ INFO ] PERFORMANCE_HINT: THROUGHPUT [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 [ INFO ] ENABLE_CPU_PINNING: True @@ -252,17 +280,25 @@ as an example. It is recommended to tune based on your applications. [ INFO ] Fill input 'token_type_ids' with random values [Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration) [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 27.56 ms + + +.. parsed-literal:: + + [ INFO ] First inference took 27.61 ms + + +.. parsed-literal:: + [Step 11/11] Dumping statistics report [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 8952 iterations - [ INFO ] Duration: 60029.00 ms + [ INFO ] Count: 8900 iterations + [ INFO ] Duration: 60039.72 ms [ INFO ] Latency: - [ INFO ] Median: 26.46 ms - [ INFO ] Average: 26.52 ms - [ INFO ] Min: 25.37 ms - [ INFO ] Max: 40.49 ms - [ INFO ] Throughput: 149.13 FPS + [ INFO ] Median: 26.68 ms + [ INFO ] Average: 26.74 ms + [ INFO ] Min: 25.09 ms + [ INFO ] Max: 39.62 ms + [ INFO ] Throughput: 148.24 FPS Benchmark quantized sparse inference performance @@ -308,18 +344,26 @@ for which a layer will be enabled. [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 [ INFO ] [ INFO ] Device info: + + +.. parsed-literal:: + [ INFO ] CPU - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 61.56 ms + + +.. parsed-literal:: + + [ INFO ] Read model took 67.79 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?] @@ -330,7 +374,11 @@ for which a layer will be enabled. [Step 5/11] Resizing model to match image sizes and given batch [ INFO ] Model batch size: 1 [ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64] - [ INFO ] Reshape model took 24.68 ms + + +.. parsed-literal:: + + [ INFO ] Reshape model took 23.92 ms [Step 6/11] Configuring input of the model [ INFO ] Model inputs: [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64] @@ -339,45 +387,20 @@ for which a layer will be enabled. [ INFO ] Model outputs: [ INFO ] logits (node: logits) : f32 / [...] / [1,2] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 1029.24 ms - [Step 8/11] Querying optimal runtime parameters - [ INFO ] Model: - [ INFO ] NETWORK_NAME: torch_jit - [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4 - [ INFO ] NUM_STREAMS: 4 - [ INFO ] AFFINITY: Affinity.CORE - [ INFO ] INFERENCE_NUM_THREADS: 4 - [ INFO ] PERF_COUNT: False - [ INFO ] INFERENCE_PRECISION_HINT: - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT - [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE - [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 - [ INFO ] ENABLE_CPU_PINNING: True - [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE - [ INFO ] ENABLE_HYPER_THREADING: True - [ INFO ] EXECUTION_DEVICES: ['CPU'] - [ INFO ] CPU_DENORMALS_OPTIMIZATION: False - [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 0.75 - [Step 9/11] Creating infer requests and preparing input tensors - [ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values! - [ WARNING ] No input files were given for input 'attention_mask'!. This input will be filled with random values! - [ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values! - [ INFO ] Fill input 'input_ids' with random values - [ INFO ] Fill input 'attention_mask' with random values - [ INFO ] Fill input 'token_type_ids' with random values - [Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration) - [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 29.95 ms - [Step 11/11] Dumping statistics report - [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 8984 iterations - [ INFO ] Duration: 60026.81 ms - [ INFO ] Latency: - [ INFO ] Median: 26.52 ms - [ INFO ] Average: 26.59 ms - [ INFO ] Min: 23.89 ms - [ INFO ] Max: 40.95 ms - [ INFO ] Throughput: 149.67 FPS + [ ERROR ] Exception from src/inference/src/core.cpp:99: + [ GENERAL_ERROR ] Exception from src/plugins/intel_cpu/src/config.cpp:158: + Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers + + Traceback (most recent call last): + File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 408, in main + compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config) + File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/runtime/ie_api.py", line 547, in compile_model + super().compile_model(model, device_name, {} if config is None else config), + RuntimeError: Exception from src/inference/src/core.cpp:99: + [ GENERAL_ERROR ] Exception from src/plugins/intel_cpu/src/config.cpp:158: + Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers + + When this might be helpful @@ -394,6 +417,6 @@ For more details about asynchronous inference with OpenVINO, refer to the following documentation: - `Deployment Optimization - Guide `__ + Guide `__ - `Inference Request - API `__ + API `__ diff --git a/docs/notebooks/117-model-server-with-output.rst b/docs/notebooks/117-model-server-with-output.rst index 3ffe8005c92..0b412ccac64 100644 --- a/docs/notebooks/117-model-server-with-output.rst +++ b/docs/notebooks/117-model-server-with-output.rst @@ -34,18 +34,18 @@ deployment: ovms_diagram -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Serving with OpenVINO Model Server <#serving-with-openvino-model-server>`__ -- `Step 1: Prepare Docker <#step--prepare-docker>`__ +- `Step 1: Prepare Docker <#step-1-prepare-docker>`__ - `Step 2: Preparing a Model - Repository <#step--preparing-a-model-repository>`__ + Repository <#step-2-preparing-a-model-repository>`__ - `Step 3: Start the Model Server - Container <#step--start-the-model-server-container>`__ + Container <#step-3-start-the-model-server-container>`__ - `Step 4: Prepare the Example Client - Components <#step--prepare-the-example-client-components>`__ + Components <#step-4-prepare-the-example-client-components>`__ - `Prerequisites <#prerequisites>`__ - `Imports <#imports>`__ @@ -58,15 +58,16 @@ deployment: - `References <#references>`__ -Serving with OpenVINO Model Server ----------------------------------------------------------------------------- +Serving with OpenVINO Model Server +---------------------------------- -OpenVINO Model Server (OVMS) is a high-performance system for serving -models. Implemented in C++ for scalability and optimized for deployment -on Intel architectures, the model server uses the same architecture and -API as TensorFlow Serving and KServe while applying OpenVINO for -inference execution. Inference service is provided via gRPC or REST API, -making deploying new algorithms and AI experiments easy. +OpenVINO Model Server (OVMS) is +a high-performance system for serving models. Implemented in C++ for +scalability and optimized for deployment on Intel architectures, the +model server uses the same architecture and API as TensorFlow Serving +and KServe while applying OpenVINO for inference execution. Inference +service is provided via gRPC or REST API, making deploying new +algorithms and AI experiments easy. .. figure:: https://user-images.githubusercontent.com/91237924/215658767-0e0fc221-aed0-4db1-9a82-6be55f244dba.png :alt: ovms_high_level @@ -75,11 +76,11 @@ making deploying new algorithms and AI experiments easy. To quickly start using OpenVINO™ Model Server, follow these steps: -Step 1: Prepare Docker ----------------------------------------------------------------- +Step 1: Prepare Docker +---------------------- -Install `Docker Engine `__, -including its +Install `Docker +Engine `__, including its `post-installation `__ steps, on your development system. To verify installation, test it, using the following command. When it is ready, it will display a test @@ -92,10 +93,10 @@ image and a message. .. parsed-literal:: - + Hello from Docker! This message shows that your installation appears to be working correctly. - + To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. @@ -104,23 +105,24 @@ image and a message. executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. - + To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash - + Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ - + For more examples and ideas, visit: https://docs.docker.com/get-started/ - -Step 2: Preparing a Model Repository ------------------------------------------------------------------------------- -The models need to be placed and mounted in a particular directory -structure and according to the following rules: +Step 2: Preparing a Model Repository +------------------------------------ + +The models need to be placed +and mounted in a particular directory structure and according to the +following rules: :: @@ -173,7 +175,7 @@ structure and according to the following rules: .. code:: ipython3 import os - + # Fetch `notebook_utils` module import urllib.request urllib.request.urlretrieve( @@ -181,40 +183,76 @@ structure and according to the following rules: filename='notebook_utils.py' ) from notebook_utils import download_file - + dedicated_dir = "models" model_name = "detection" model_version = "1" - + MODEL_DIR = f"{dedicated_dir}/{model_name}/{model_version}" XML_PATH = "horizontal-text-detection-0001.xml" BIN_PATH = "horizontal-text-detection-0001.bin" os.makedirs(MODEL_DIR, exist_ok=True) model_xml_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.xml" model_bin_url = "https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/horizontal-text-detection-0001/FP32/horizontal-text-detection-0001.bin" - + download_file(model_xml_url, XML_PATH, MODEL_DIR) download_file(model_bin_url, BIN_PATH, MODEL_DIR) + .. parsed-literal:: - Model Copied to "./models/detection/1". + models/detection/1/horizontal-text-detection-0001.xml: 0%| | 0.00/680k [00:009000/tcp, :::9000->9000/tcp ovms + 64aa9391ba01 openvino/model_server:latest "/ovms/bin/ovms --mo…" 29 seconds ago Up 28 seconds 0.0.0.0:37581->9000/tcp, :::37581->9000/tcp ovms The required Model Server parameters are listed below. For additional configuration options, see the `Model Server Parameters -section `__. +section `__. .. raw:: html @@ -648,22 +686,24 @@ openvino/model_server:latest -If the serving port ``9000`` is already in use, please switch it to -another available port on your system. For example:\ ``-p 9020:9000`` +If the serving port is already in use, please switch it to another +available port on your system. For example:\ ``-p 9020:9000`` -Step 4: Prepare the Example Client Components ---------------------------------------------------------------------------------------- +Step 4: Prepare the Example Client Components +--------------------------------------------- + +OpenVINO Model Server exposes +two sets of APIs: one compatible with ``TensorFlow Serving`` and another +one, with ``KServe API``, for inference. Both APIs work on ``gRPC`` and +``REST``\ interfaces. Supporting two sets of APIs makes OpenVINO Model +Server easier to plug into existing systems the already leverage one of +these APIs for inference. This example will demonstrate how to write a +TensorFlow Serving API client for object detection. + +Prerequisites +~~~~~~~~~~~~~ -OpenVINO Model Server exposes two sets of APIs: one compatible with -``TensorFlow Serving`` and another one, with ``KServe API``, for -inference. Both APIs work on ``gRPC`` and ``REST``\ interfaces. -Supporting two sets of APIs makes OpenVINO Model Server easier to plug -into existing systems the already leverage one of these APIs for -inference. This example will demonstrate how to write a TensorFlow -Serving API client for object detection. -Prerequisites -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Install necessary packages. @@ -674,31 +714,13 @@ Install necessary packages. .. parsed-literal:: - Collecting ovmsclient - Downloading ovmsclient-2022.3-py3-none-any.whl (163 kB) -  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.0/164.0 KB 2.1 MB/s eta 0:00:00a 0:00:01 - Requirement already satisfied: numpy>=1.16.6 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (1.23.4) - Requirement already satisfied: requests>=2.27.1 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (2.27.1) - Collecting grpcio>=1.47.0 - Downloading grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB) -  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 5.6 MB/s eta 0:00:0000:0100:01 - Requirement already satisfied: protobuf>=3.19.4 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from ovmsclient) (3.19.6) - Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (1.26.9) - Requirement already satisfied: idna<4,>=2.5 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (3.3) - Requirement already satisfied: certifi>=2017.4.17 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (2021.10.8) - Requirement already satisfied: charset-normalizer~=2.0.0 in /home/adrian/repos/openvino_notebooks_adrian/venv/lib/python3.9/site-packages (from requests>=2.27.1->ovmsclient) (2.0.12) - Installing collected packages: grpcio, ovmsclient - Attempting uninstall: grpcio - Found existing installation: grpcio 1.34.1 - Uninstalling grpcio-1.34.1: - Successfully uninstalled grpcio-1.34.1 - Successfully installed grpcio-1.51.3 ovmsclient-2022.3 - WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available. - You should consider upgrading via the '/home/adrian/repos/openvino_notebooks_adrian/venv/bin/python -m pip install --upgrade pip' command. - + Note: you may need to restart the kernel to use updated packages. + + +Imports +~~~~~~~ + -Imports -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 @@ -707,13 +729,15 @@ Imports import matplotlib.pyplot as plt from ovmsclient import make_grpc_client -Request Model Status -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Request Model Status +~~~~~~~~~~~~~~~~~~~~ + + .. code:: ipython3 - address = "localhost:9000" - + address = "localhost:" + str(port) + # Bind the grpc address to the client object client = make_grpc_client(address) model_status = client.get_model_status(model_name=model_name) @@ -725,8 +749,10 @@ Request Model Status {1: {'state': 'AVAILABLE', 'error_code': 0, 'error_message': 'OK'}} -Request Model Metadata -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Request Model Metadata +~~~~~~~~~~~~~~~~~~~~~~ + + .. code:: ipython3 @@ -736,11 +762,13 @@ Request Model Metadata .. parsed-literal:: - {'model_version': 1, 'inputs': {'image': {'shape': [1, 3, 704, 704], 'dtype': 'DT_FLOAT'}}, 'outputs': {'1469_1470.0': {'shape': [-1], 'dtype': 'DT_FLOAT'}, '1078_1079.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1330_1331.0': {'shape': [36], 'dtype': 'DT_FLOAT'}, 'labels': {'shape': [-1], 'dtype': 'DT_INT32'}, '1267_1268.0': {'shape': [121], 'dtype': 'DT_FLOAT'}, '1141_1142.0': {'shape': [1000], 'dtype': 'DT_FLOAT'}, '1204_1205.0': {'shape': [484], 'dtype': 'DT_FLOAT'}, 'boxes': {'shape': [-1, 5], 'dtype': 'DT_FLOAT'}}} + {'model_version': 1, 'inputs': {'image': {'shape': [1, 3, 704, 704], 'dtype': 'DT_FLOAT'}}, 'outputs': {'boxes': {'shape': [-1, 5], 'dtype': 'DT_FLOAT'}, 'labels': {'shape': [-1], 'dtype': 'DT_INT64'}}} + + +Load input image +~~~~~~~~~~~~~~~~ -Load input image -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 @@ -749,43 +777,51 @@ Load input image "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/intel_rnb.jpg", directory="data" ) - + # Text detection models expect an image in BGR format. image = cv2.imread(str(image_filename)) fp_image = image.astype("float32") - + # Resize the image to meet network expected input sizes. input_shape = model_metadata['inputs']['image']['shape'] height, width = input_shape[2], input_shape[3] resized_image = cv2.resize(fp_image, (height, width)) - + # Reshape to the network input shape. input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0) plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) +.. parsed-literal:: + + data/intel_rnb.jpg: 0%| | 0.00/288k [00:00 + -.. image:: 117-model-server-with-output_files/117-model-server-with-output_21_1.png +.. image:: 117-model-server-with-output_files/117-model-server-with-output_23_2.png + + +Request Prediction on a Numpy Array +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Request Prediction on a Numpy Array -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 inputs = {"image": input_image} - + # Run inference on model server and receive the result data boxes = client.predict(inputs=inputs, model_name=model_name)['boxes'] - + # Remove zero only boxes. boxes = boxes[~np.all(boxes == 0, axis=1)] print(boxes) @@ -793,16 +829,18 @@ Request Prediction on a Numpy Array .. parsed-literal:: - [[3.9992419e+02 8.1032524e+01 5.6187299e+02 1.3619952e+02 5.3706491e-01] - [2.6189725e+02 6.8310547e+01 3.8541251e+02 1.2095630e+02 4.7559953e-01] - [6.1644586e+02 2.8008759e+02 6.6627545e+02 3.1178854e+02 4.4982004e-01] - [2.0762042e+02 6.2798470e+01 2.3444728e+02 1.0706525e+02 3.7216505e-01] - [5.1742780e+02 5.5603595e+02 5.4927539e+02 5.8736023e+02 3.2588077e-01] - [2.2261986e+01 4.5406548e+01 1.8868817e+02 1.0225631e+02 3.0407205e-01]] + [[4.0075238e+02 8.1240105e+01 5.6262683e+02 1.3609659e+02 5.3646392e-01] + [2.6150497e+02 6.8225861e+01 3.8433078e+02 1.2111545e+02 4.7504124e-01] + [6.1611401e+02 2.8000638e+02 6.6605963e+02 3.1116574e+02 4.5030469e-01] + [2.0762566e+02 6.2619057e+01 2.3446707e+02 1.0711832e+02 3.7426147e-01] + [5.1753296e+02 5.5611102e+02 5.4918005e+02 5.8740009e+02 3.2477754e-01] + [2.2038467e+01 4.5390991e+01 1.8856328e+02 1.0215196e+02 2.9959568e-01]] + + +Visualization +~~~~~~~~~~~~~ -Visualization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 @@ -811,31 +849,31 @@ Visualization def convert_result_to_image(bgr_image, resized_image, boxes, threshold=0.3, conf_labels=True): # Define colors for boxes and descriptions. colors = {"red": (255, 0, 0), "green": (0, 255, 0)} - + # Fetch the image shapes to calculate a ratio. (real_y, real_x), (resized_y, resized_x) = bgr_image.shape[:2], resized_image.shape[:2] ratio_x, ratio_y = real_x / resized_x, real_y / resized_y - + # Convert the base image from BGR to RGB format. rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB) - + # Iterate through non-zero boxes. for box in boxes: # Pick a confidence factor from the last place in an array. conf = box[-1] if conf > threshold: # Convert float to int and multiply corner position of each box by x and y ratio. - # If the bounding box is found at the top of the image, - # position the upper box bar little lower to make it visible on the image. + # If the bounding box is found at the top of the image, + # position the upper box bar little lower to make it visible on the image. (x_min, y_min, x_max, y_max) = [ - int(max(corner_position * ratio_y, 10)) if idx % 2 + int(max(corner_position * ratio_y, 10)) if idx % 2 else int(corner_position * ratio_x) for idx, corner_position in enumerate(box[:-1]) ] - + # Draw a box based on the position, parameters in rectangle function are: image, start_point, end_point, color, thickness. rgb_image = cv2.rectangle(rgb_image, (x_min, y_min), (x_max, y_max), colors["green"], 3) - + # Add text to the image based on position and confidence. # Parameters in text function are: image, text, bottom-left_corner_textfield, font, font_scale, color, thickness, line_type. if conf_labels: @@ -849,7 +887,7 @@ Visualization 1, cv2.LINE_AA, ) - + return rgb_image .. code:: ipython3 @@ -863,12 +901,12 @@ Visualization .. parsed-literal:: - + -.. image:: 117-model-server-with-output_files/117-model-server-with-output_26_1.png +.. image:: 117-model-server-with-output_files/117-model-server-with-output_28_1.png To stop and remove the model server container, you can use the following @@ -884,10 +922,12 @@ command: ovms -References ----------------------------------------------------- +References +---------- + + 1. `OpenVINO™ Model Server - documentation `__ + documentation `__ 2. `OpenVINO™ Model Server GitHub repository `__ diff --git a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_21_1.png b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_21_1.png deleted file mode 100644 index 5c7d535ce89..00000000000 --- a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_21_1.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e20e0cc48768c8a971c0e0cd72666c63f0bd5940256a1d43b0a2de2f7874eeef -size 112408 diff --git a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_23_2.png b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_23_2.png new file mode 100644 index 00000000000..405f2240ea5 --- /dev/null +++ b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_23_2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:151f6c5d9e90a68411adce7359011fe6580a071decbc7f1cacbf60975083bc9d +size 305482 diff --git a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_26_1.png b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_26_1.png deleted file mode 100644 index 5631db94bbb..00000000000 --- a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_26_1.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:df4dbdbd56ef78ad7619bfb7fcfca6c96dccce598b3a5263486387b186e8d08f -size 232667 diff --git a/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_28_1.png b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_28_1.png new file mode 100644 index 00000000000..74a53ee7ee0 --- /dev/null +++ b/docs/notebooks/117-model-server-with-output_files/117-model-server-with-output_28_1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c0dd02ecae6935c5d65ae1ad543ef6a918185c440b667e68b98f4bd681762030 +size 458294 diff --git a/docs/notebooks/117-model-server-with-output_files/index.html b/docs/notebooks/117-model-server-with-output_files/index.html deleted file mode 100644 index 0df2967d038..00000000000 --- a/docs/notebooks/117-model-server-with-output_files/index.html +++ /dev/null @@ -1,8 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/117-model-server-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231030220807/dist/rst_files/117-model-server-with-output_files/


../
-117-model-server-with-output_21_1.png              31-Oct-2023 00:35              112408
-117-model-server-with-output_26_1.png              31-Oct-2023 00:35              232667
-

- diff --git a/docs/notebooks/117-model-server-with-output_files/index.html.tmp b/docs/notebooks/117-model-server-with-output_files/index.html.tmp new file mode 100644 index 00000000000..2074b102cd1 --- /dev/null +++ b/docs/notebooks/117-model-server-with-output_files/index.html.tmp @@ -0,0 +1,8 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/117-model-server-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/117-model-server-with-output_files/


../
+117-model-server-with-output_23_2.png              26-Jan-2024 01:04              305482
+117-model-server-with-output_28_1.png              26-Jan-2024 01:04              458294
+

+ diff --git a/docs/notebooks/118-optimize-preprocessing-with-output.rst b/docs/notebooks/118-optimize-preprocessing-with-output.rst index bf39ecd4881..7bf44e38ce9 100644 --- a/docs/notebooks/118-optimize-preprocessing-with-output.rst +++ b/docs/notebooks/118-optimize-preprocessing-with-output.rst @@ -23,7 +23,8 @@ This tutorial include following steps: - Comparing results on one picture. - Comparing performance. -**Table of contents:** +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Settings <#settings>`__ - `Imports <#imports>`__ @@ -39,7 +40,7 @@ This tutorial include following steps: - `Convert model to OpenVINO IR with model conversion API <#convert-model-to-openvino-ir-with-model-conversion-api>`__ - - `Create PrePostProcessor + - `Create ``PrePostProcessor`` Object <#create-prepostprocessor-object>`__ - `Declare User’s Data Format <#declare-users-data-format>`__ - `Declaring Model Layout <#declaring-model-layout>`__ @@ -105,10 +106,14 @@ Imports .. parsed-literal:: - 2023-11-14 23:00:32.637266: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-11-14 23:00:32.671311: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + 2024-01-25 23:07:18.341652: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 23:07:18.375926: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-11-14 23:00:33.179278: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + 2024-01-25 23:07:18.888977: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Setup image and device @@ -189,12 +194,12 @@ and save it to the disk. .. parsed-literal:: - 2023-11-14 23:00:37.345835: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW - 2023-11-14 23:00:37.345869: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07 - 2023-11-14 23:00:37.345874: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07 - 2023-11-14 23:00:37.346012: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2 - 2023-11-14 23:00:37.346027: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3 - 2023-11-14 23:00:37.346030: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration + 2024-01-25 23:07:22.553946: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW + 2024-01-25 23:07:22.553985: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07 + 2024-01-25 23:07:22.553989: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07 + 2024-01-25 23:07:22.554129: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2 + 2024-01-25 23:07:22.554144: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3 + 2024-01-25 23:07:22.554147: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration .. parsed-literal:: @@ -268,7 +273,7 @@ Graph modifications of a model shall be performed after the model is read from a drive and before it is loaded on the actual device. Pre-processing support following operations (please, see more details -`here `__) +`here `__) - Mean/Scale Normalization - Converting Precision @@ -304,7 +309,7 @@ Create ``PrePostProcessor`` Object The -`PrePostProcessor() `__ +`PrePostProcessor() `__ class enables specifying the preprocessing and postprocessing steps for a model. @@ -329,7 +334,7 @@ about user’s input tensor will be initialized to same data (type/shape/etc) as model’s input parameter. User application can override particular parameters according to application’s data. Refer to the following -`page `__ +`page `__ for more information about parameters for overriding. Below is all the specified input information: @@ -355,7 +360,7 @@ for mean/scale normalization. .. parsed-literal:: - + @@ -367,7 +372,7 @@ Declaring Model Layout Model input already has information about precision and shape. Preprocessing API is not intended to modify this. The only thing that may be specified is input data -`layout `__. +`layout `__. .. code:: ipython3 @@ -386,7 +391,7 @@ may be specified is input data .. parsed-literal:: - + @@ -406,7 +411,7 @@ Perform the following: dynamic size, for example, ``{?, 3, ?, ?}`` resize will not know how to resize the picture. Therefore, in this case, target height/ width should be specified. For more details, see also the - `PreProcessSteps.resize() `__. + `PreProcessSteps.resize() `__. - Subtract mean from each channel. - Divide each pixel data to appropriate scale value. @@ -427,7 +432,7 @@ then such conversion will be added explicitly. .. parsed-literal:: - + @@ -646,6 +651,10 @@ Compare performance .. parsed-literal:: - IR model in OpenVINO Runtime/CPU with manual image preprocessing: 0.0152 seconds per image, FPS: 65.58 - IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0187 seconds per image, FPS: 53.52 + IR model in OpenVINO Runtime/CPU with manual image preprocessing: 0.0149 seconds per image, FPS: 67.09 + + +.. parsed-literal:: + + IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0182 seconds per image, FPS: 54.89 diff --git a/docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_14_1.png b/docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_14_1.png index 15c5ba574c1..3f302494abc 100644 --- a/docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_14_1.png +++ b/docs/notebooks/118-optimize-preprocessing-with-output_files/118-optimize-preprocessing-with-output_14_1.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7511b8a4e5b047600d5fed14fbc7e9653a868bc5253abf1e0c3ef649b47bc408 +oid sha256:2dd4338c6c163e7693885ce544e8c9cd2aecedf3b136fa295e22877f37b5634c size 387941 diff --git a/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html b/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html deleted file mode 100644 index 77439bd3f11..00000000000 --- a/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html +++ /dev/null @@ -1,7 +0,0 @@ - -Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/118-optimize-preprocessing-with-output_files/ - -

Index of /projects/ov-notebook/0.1.0-latest/20231114220808/dist/rst_files/118-optimize-preprocessing-with-output_files/


../
-118-optimize-preprocessing-with-output_14_1.png    15-Nov-2023 00:43              387941
-

- diff --git a/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html.tmp b/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html.tmp new file mode 100644 index 00000000000..bd254a5f87c --- /dev/null +++ b/docs/notebooks/118-optimize-preprocessing-with-output_files/index.html.tmp @@ -0,0 +1,7 @@ + +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/118-optimize-preprocessing-with-output_files/ + +

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/118-optimize-preprocessing-with-output_files/


../
+118-optimize-preprocessing-with-output_14_1.png    26-Jan-2024 01:04              387941
+

+ diff --git a/docs/notebooks/119-tflite-to-openvino-with-output.rst b/docs/notebooks/119-tflite-to-openvino-with-output.rst index a80be0f9317..28ea431fe3b 100644 --- a/docs/notebooks/119-tflite-to-openvino-with-output.rst +++ b/docs/notebooks/119-tflite-to-openvino-with-output.rst @@ -14,8 +14,8 @@ IR, load the model in `OpenVINO Runtime `__ and do inference with a sample image. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Preparation <#preparation>`__ @@ -47,7 +47,7 @@ Install requirements %pip install -q "openvino>=2023.1.0" %pip install -q opencv-python requests tqdm - + # Fetch `notebook_utils` module import urllib.request urllib.request.urlretrieve( @@ -59,6 +59,10 @@ Install requirements .. parsed-literal:: Note: you may need to restart the kernel to use updated packages. + + +.. parsed-literal:: + Note: you may need to restart the kernel to use updated packages. @@ -73,7 +77,7 @@ Imports import numpy as np from PIL import Image import openvino as ov - + from notebook_utils import download_file, load_image Download TFLite model @@ -85,10 +89,10 @@ Download TFLite model model_dir = Path("model") tflite_model_path = model_dir / "efficientnet_lite0_fp32_2.tflite" - + ov_model_path = tflite_model_path.with_suffix(".xml") model_url = "https://www.kaggle.com/models/tensorflow/efficientnet/frameworks/tfLite/variations/lite0-fp32/versions/2?lite-format=tflite" - + download_file(model_url, tflite_model_path.name, model_dir) @@ -102,7 +106,7 @@ Download TFLite model .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/119-tflite-to-openvino/model/efficientnet_lite0_fp32_2.tflite') @@ -122,7 +126,7 @@ serialization by ``ov.save_model``. For more information about model conversion, see this `page `__. For TensorFlow Lite models support, refer to this -`tutorial `__. +`tutorial `__. .. code:: ipython3 @@ -144,12 +148,12 @@ Load model using OpenVINO TensorFlow Lite Frontend TensorFlow Lite models are supported via ``FrontEnd`` API. You may skip conversion to IR and read models directly by OpenVINO runtime API. For more examples supported formats reading via Frontend API, please look -this `tutorial <../002-openvino-api>`__. +this `tutorial <002-openvino-api-with-output.html>`__. .. code:: ipython3 core = ov.Core() - + ov_model = core.read_model(tflite_model_path) Run OpenVINO model inference @@ -179,14 +183,14 @@ select device from dropdown list for running inference using OpenVINO .. code:: ipython3 import ipywidgets as widgets - + device = widgets.Dropdown( options=core.available_devices + ["AUTO"], value='AUTO', description='Device:', disabled=False, ) - + device @@ -207,11 +211,11 @@ select device from dropdown list for running inference using OpenVINO imagenet_classes_file_path = download_file("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/datasets/imagenet/imagenet_2012.txt") imagenet_classes = open(imagenet_classes_file_path).read().splitlines() - + top1_predicted_cls_id = np.argmax(predicted_scores) top1_predicted_score = predicted_scores[0][top1_predicted_cls_id] predicted_label = imagenet_classes[top1_predicted_cls_id] - + display(image.resize((640, 512))) print(f"Predicted label: {predicted_label} with probability {top1_predicted_score :2f}") @@ -235,7 +239,7 @@ Estimate Model Performance -------------------------- `Benchmark -Tool `__ +Tool `__ is used to measure the inference performance of the model on CPU and GPU. @@ -258,22 +262,26 @@ GPU. .. parsed-literal:: Benchmark model inference on CPU + + +.. parsed-literal:: + [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] [ INFO ] Device info: [ INFO ] CPU - [ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD - [ INFO ] - [ INFO ] + [ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3 + [ INFO ] + [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT. [Step 4/11] Reading model files [ INFO ] Loading model files - [ INFO ] Read model took 28.35 ms + [ INFO ] Read model took 10.06 ms [ INFO ] Original model I/O parameters: [ INFO ] Model inputs: [ INFO ] images (node: images) : f32 / [...] / [1,224,224,3] @@ -287,7 +295,11 @@ GPU. [ INFO ] Model outputs: [ INFO ] Softmax (node: 63) : f32 / [...] / [1,1000] [Step 7/11] Loading the model to the device - [ INFO ] Compile model took 147.48 ms + + +.. parsed-literal:: + + [ INFO ] Compile model took 138.70 ms [Step 8/11] Querying optimal runtime parameters [ INFO ] Model: [ INFO ] NETWORK_NAME: TensorFlow_Lite_Frontend_IR @@ -295,9 +307,9 @@ GPU. [ INFO ] NUM_STREAMS: 6 [ INFO ] AFFINITY: Affinity.CORE [ INFO ] INFERENCE_NUM_THREADS: 24 - [ INFO ] PERF_COUNT: False + [ INFO ] PERF_COUNT: NO [ INFO ] INFERENCE_PRECISION_HINT: - [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT + [ INFO ] PERFORMANCE_HINT: THROUGHPUT [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0 [ INFO ] ENABLE_CPU_PINNING: True @@ -308,18 +320,22 @@ GPU. [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0 [Step 9/11] Creating infer requests and preparing input tensors [ WARNING ] No input files were given for input 'images'!. This input will be filled with random values! - [ INFO ] Fill input 'images' with random values + [ INFO ] Fill input 'images' with random values [Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration) [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop). - [ INFO ] First inference took 7.23 ms + [ INFO ] First inference took 7.39 ms + + +.. parsed-literal:: + [Step 11/11] Dumping statistics report [ INFO ] Execution Devices:['CPU'] - [ INFO ] Count: 17526 iterations - [ INFO ] Duration: 15005.83 ms + [ INFO ] Count: 17544 iterations + [ INFO ] Duration: 15006.83 ms [ INFO ] Latency: - [ INFO ] Median: 5.00 ms + [ INFO ] Median: 5.01 ms [ INFO ] Average: 5.00 ms - [ INFO ] Min: 2.72 ms - [ INFO ] Max: 15.43 ms - [ INFO ] Throughput: 1167.95 FPS + [ INFO ] Min: 2.90 ms + [ INFO ] Max: 14.27 ms + [ INFO ] Throughput: 1169.07 FPS diff --git a/docs/notebooks/119-tflite-to-openvino-with-output_files/index.html b/docs/notebooks/119-tflite-to-openvino-with-output_files/index.html.tmp similarity index 50% rename from docs/notebooks/119-tflite-to-openvino-with-output_files/index.html rename to docs/notebooks/119-tflite-to-openvino-with-output_files/index.html.tmp index ae8ae03dd1a..97316fb93fc 100644 --- a/docs/notebooks/119-tflite-to-openvino-with-output_files/index.html +++ b/docs/notebooks/119-tflite-to-openvino-with-output_files/index.html.tmp @@ -1,8 +1,8 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/119-tflite-to-openvino-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/119-tflite-to-openvino-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/119-tflite-to-openvino-with-output_files/


../
-119-tflite-to-openvino-with-output_16_1.jpg        07-Dec-2023 00:49               68170
-119-tflite-to-openvino-with-output_16_1.png        07-Dec-2023 00:49              621006
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/119-tflite-to-openvino-with-output_files/


../
+119-tflite-to-openvino-with-output_16_1.jpg        26-Jan-2024 01:05               68170
+119-tflite-to-openvino-with-output_16_1.png        26-Jan-2024 01:05              621006
 

diff --git a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output.rst b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output.rst index 347579959ec..38a836e7946 100644 --- a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output.rst +++ b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output.rst @@ -18,11 +18,11 @@ Inception ResNet V2 `__ instance segmentation model to OpenVINO `Intermediate Representation `__ -(OpenVINO IR) format, using `Model -Optimizer `__. +(OpenVINO IR) format, using `Model Conversion +API `__. After creating the OpenVINO IR, load the model in `OpenVINO Runtime `__ -and do inference with a sample image. +and do inference with a sample image. **Table of contents:** @@ -66,7 +66,7 @@ The notebook uses utility functions. The cell below will download the # Fetch the notebook utils script from the openvino_notebooks repo import urllib.request - + urllib.request.urlretrieve( url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py", filename="notebook_utils.py", @@ -81,15 +81,15 @@ Imports # Standard python modules from pathlib import Path - + # External modules and dependencies import cv2 import matplotlib.pyplot as plt import numpy as np - + # Notebook utils module from notebook_utils import download_file - + # OpenVINO modules import openvino as ov @@ -105,21 +105,21 @@ Define model related variables and create corresponding directories: # Create directories for models files model_dir = Path("model") model_dir.mkdir(exist_ok=True) - + # Create directory for TensorFlow model tf_model_dir = model_dir / "tf" tf_model_dir.mkdir(exist_ok=True) - + # Create directory for OpenVINO IR model ir_model_dir = model_dir / "ir" ir_model_dir.mkdir(exist_ok=True) - + model_name = "mask_rcnn_inception_resnet_v2_1024x1024" - + openvino_ir_path = ir_model_dir / f"{model_name}.xml" - + tf_model_url = "https://www.kaggle.com/models/tensorflow/mask-rcnn-inception-resnet-v2/frameworks/tensorFlow2/variations/1024x1024/versions/1?tf-hub-format=compressed" - + tf_model_archive_filename = f"{model_name}.tar.gz" Download Model from TensorFlow Hub @@ -152,7 +152,7 @@ archive: .. code:: ipython3 import tarfile - + with tarfile.open(tf_model_dir / tf_model_archive_filename) as file: file.extractall(path=tf_model_dir) @@ -180,7 +180,7 @@ when the model is run in the future. .. code:: ipython3 ov_model = ov.convert_model(tf_model_dir) - + # Save converted OpenVINO IR model to the corresponding directory ov.save_model(ov_model, openvino_ir_path) @@ -199,7 +199,7 @@ select device from dropdown list for running inference using OpenVINO .. code:: ipython3 import ipywidgets as widgets - + core = ov.Core() device = widgets.Dropdown( options=core.available_devices + ["AUTO"], @@ -207,7 +207,7 @@ select device from dropdown list for running inference using OpenVINO description='Device:', disabled=False, ) - + device @@ -263,12 +263,12 @@ the first (and highest) detection score. model_inputs = compiled_model.inputs model_outputs = compiled_model.outputs - + print("Model inputs count:", len(model_inputs)) print("Model inputs:") for _input in model_inputs: print(" ", _input) - + print("Model outputs count:", len(model_outputs)) print("Model outputs:") for output in model_outputs: @@ -317,7 +317,7 @@ Load and save an image: .. code:: ipython3 image_path = Path("./data/coco_bike.jpg") - + download_file( url="https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg", filename=image_path.name, @@ -337,16 +337,16 @@ Read the image, resize and convert it to the input shape of the network: # Read the image image = cv2.imread(filename=str(image_path)) - + # The network expects images in RGB format image = cv2.cvtColor(image, code=cv2.COLOR_BGR2RGB) - + # Resize the image to the network input shape resized_image = cv2.resize(src=image, dsize=(255, 255)) - + # Add batch dimension to image network_input_image = np.expand_dims(resized_image, 0) - + # Show the image plt.imshow(image) @@ -355,7 +355,7 @@ Read the image, resize and convert it to the input shape of the network: .. parsed-literal:: - + @@ -382,23 +382,23 @@ be extracted from the result. For further model result visualization detection_boxes = compiled_model.output("detection_boxes") image_detection_boxes = inference_result[detection_boxes] print("image_detection_boxes:", image_detection_boxes.shape) - + detection_masks = compiled_model.output("detection_masks") image_detection_masks = inference_result[detection_masks] print("image_detection_masks:", image_detection_masks.shape) - + detection_classes = compiled_model.output("detection_classes") image_detection_classes = inference_result[detection_classes] print("image_detection_classes:", image_detection_classes.shape) - + detection_scores = compiled_model.output("detection_scores") image_detection_scores = inference_result[detection_scores] print("image_detection_scores:", image_detection_scores.shape) - + num_detections = compiled_model.output("num_detections") image_num_detections = inference_result[num_detections] print("image_detections_num:", image_num_detections) - + # Alternatively, inference result data can be extracted by model output name with `.get()` method assert (inference_result[detection_boxes] == inference_result.get("detection_boxes")).all(), "extracted inference result data should be equal" @@ -423,14 +423,14 @@ Define utility functions to visualize the inference results import random from typing import Optional - - + + def add_detection_box( box: np.ndarray, image: np.ndarray, mask: np.ndarray, label: Optional[str] = None ) -> np.ndarray: """ Helper function for adding single bounding box to the image - + Parameters ---------- box : np.ndarray @@ -441,18 +441,18 @@ Define utility functions to visualize the inference results Segmentation mask in format (H, W) label : str, optional Detection box label string, if not provided will not be added to result image (default is None) - + Returns ------- np.ndarray NumPy array including image, detection box, and segmentation mask - + """ ymin, xmin, ymax, xmax = box point1, point2 = (int(xmin), int(ymin)), (int(xmax), int(ymax)) box_color = [random.randint(0, 255) for _ in range(3)] line_thickness = round(0.002 * (image.shape[0] + image.shape[1]) / 2) + 1 - + result = cv2.rectangle( img=image, pt1=point1, @@ -461,7 +461,7 @@ Define utility functions to visualize the inference results thickness=line_thickness, lineType=cv2.LINE_AA, ) - + if label: font_thickness = max(line_thickness - 1, 1) font_face = 0 @@ -504,12 +504,12 @@ Define utility functions to visualize the inference results def get_mask_frame(box, frame, mask): """ Transform a binary mask to fit within a specified bounding box in a frame using perspective transformation. - + Args: box (tuple): A bounding box represented as a tuple (y_min, x_min, y_max, x_max). frame (numpy.ndarray): The larger frame or image where the mask will be placed. mask (numpy.ndarray): A binary mask image to be transformed. - + Returns: numpy.ndarray: A transformed mask image that fits within the specified bounding box in the frame. """ @@ -534,10 +534,10 @@ Define utility functions to visualize the inference results .. code:: ipython3 from typing import Dict - + from openvino.runtime.utils.data_helpers import OVDict - - + + def visualize_inference_result( inference_result: OVDict, image: np.ndarray, @@ -546,7 +546,7 @@ Define utility functions to visualize the inference results ): """ Helper function for visualizing inference result on the image - + Parameters ---------- inference_result : OVDict @@ -563,13 +563,13 @@ Define utility functions to visualize the inference results detection_scores = inference_result.get("detection_scores") num_detections = inference_result.get("num_detections") detection_masks = inference_result.get("detection_masks") - + detections_limit = int( min(detections_limit, num_detections[0]) if detections_limit is not None else num_detections[0] ) - + # Normalize detection boxes coordinates to original image size original_image_height, original_image_width, _ = image.shape normalized_detection_boxes = detection_boxes[0, :detections_limit] * [ @@ -589,7 +589,7 @@ Define utility functions to visualize the inference results result = add_detection_box( box=normalized_detection_boxes[i], image=result, mask=mask_reframed, label=label ) - + plt.imshow(result) TensorFlow Instance Segmentation model @@ -605,7 +605,7 @@ Zoo `__: .. code:: ipython3 coco_labels_file_path = Path("./data/coco_91cl.txt") - + download_file( url="https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/coco_91cl.txt", filename=coco_labels_file_path.name, @@ -628,7 +628,7 @@ file: with open(coco_labels_file_path, "r") as file: coco_labels = file.read().strip().split("\n") coco_labels_map = dict(enumerate(coco_labels, 1)) - + print(coco_labels_map) diff --git a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png index 99694eb8f03..176ce97e1f5 100644 --- a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png +++ b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/120-tensorflow-instance-segmentation-to-openvino-with-output_39_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1359365695e66164137690b56f261eae3364dff68f3e155a3e6b878ff17cea3f -size 393190 +oid sha256:ed008919b91d0e231d8872fb4fae77bdfc3a513547568b9448b69de0065625c5 +size 393453 diff --git a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html.tmp similarity index 62% rename from docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html rename to docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html.tmp index 9911d8830b7..9ebafdab39c 100644 --- a/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html +++ b/docs/notebooks/120-tensorflow-instance-segmentation-to-openvino-with-output_files/index.html.tmp @@ -1,8 +1,8 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/


../
-120-tensorflow-instance-segmentation-to-openvin..> 07-Dec-2023 00:49              395346
-120-tensorflow-instance-segmentation-to-openvin..> 07-Dec-2023 00:49              393190
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-instance-segmentation-to-openvino-with-output_files/


../
+120-tensorflow-instance-segmentation-to-openvin..> 26-Jan-2024 01:05              395346
+120-tensorflow-instance-segmentation-to-openvin..> 26-Jan-2024 01:05              393453
 

diff --git a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst index e4f7d5c5fdd..fcc8ba2cba6 100644 --- a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst +++ b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output.rst @@ -20,11 +20,11 @@ object detection model to OpenVINO `Intermediate Representation `__ (OpenVINO IR) format, using Model Converter. After creating the OpenVINO IR, load the model in `OpenVINO -Runtime `__ +Runtime `__ and do inference with a sample image. -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `Prerequisites <#prerequisites>`__ - `Imports <#imports>`__ @@ -74,7 +74,7 @@ The notebook uses utility functions. The cell below will download the # Fetch the notebook utils script from the openvino_notebooks repo import urllib.request - + urllib.request.urlretrieve( url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py", filename="notebook_utils.py", @@ -89,14 +89,14 @@ Imports # Standard python modules from pathlib import Path - + # External modules and dependencies import cv2 import matplotlib.pyplot as plt import numpy as np # OpenVINO import import openvino as ov - + # Notebook utils module from notebook_utils import download_file @@ -112,21 +112,21 @@ Define model related variables and create corresponding directories: # Create directories for models files model_dir = Path("model") model_dir.mkdir(exist_ok=True) - + # Create directory for TensorFlow model tf_model_dir = model_dir / "tf" tf_model_dir.mkdir(exist_ok=True) - + # Create directory for OpenVINO IR model ir_model_dir = model_dir / "ir" ir_model_dir.mkdir(exist_ok=True) - + model_name = "faster_rcnn_resnet50_v1_640x640" - + openvino_ir_path = ir_model_dir / f"{model_name}.xml" - + tf_model_url = "https://www.kaggle.com/models/tensorflow/faster-rcnn-resnet-v1/frameworks/tensorFlow2/variations/faster-rcnn-resnet50-v1-640x640/versions/1?tf-hub-format=compressed" - + tf_model_archive_filename = f"{model_name}.tar.gz" Download Model from TensorFlow Hub @@ -157,7 +157,7 @@ from TensorFlow Hub: .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/model/tf/faster_rcnn_resnet50_v1_640x640.tar.gz') @@ -166,7 +166,7 @@ Extract TensorFlow Object Detection model from the downloaded archive: .. code:: ipython3 import tarfile - + with tarfile.open(tf_model_dir / tf_model_archive_filename) as file: file.extractall(path=tf_model_dir) @@ -175,8 +175,8 @@ Convert Model to OpenVINO IR -OpenVINO Model Converter Python API can be used to convert the -TensorFlow model to OpenVINO IR. +OpenVINO Model Conversion API can be used to convert the TensorFlow +model to OpenVINO IR. ``ov.convert_model`` function accept path to TensorFlow model and returns OpenVINO Model class instance which represents this model. Also @@ -188,15 +188,15 @@ The converted model is ready to load on a device using ``compile_model`` or saved on disk using the ``save_model`` function to reduce loading time when the model is run in the future. -See the `Model Converter Developer +See the `Model Preparation Guide `__ -for more information about Model Converter and TensorFlow `models -support `__. +for more information about model conversion and TensorFlow `models +support `__. .. code:: ipython3 ov_model = ov.convert_model(tf_model_dir) - + # Save converted OpenVINO IR model to the corresponding directory ov.save_model(ov_model, openvino_ir_path) @@ -215,7 +215,7 @@ select device from dropdown list for running inference using OpenVINO .. code:: ipython3 import ipywidgets as widgets - + core = ov.Core() device = widgets.Dropdown( options=core.available_devices + ["AUTO"], @@ -223,7 +223,7 @@ select device from dropdown list for running inference using OpenVINO description='Device:', disabled=False, ) - + device @@ -290,10 +290,10 @@ for more information about model inputs, outputs and their formats. model_inputs = compiled_model.inputs model_input = compiled_model.input(0) model_outputs = compiled_model.outputs - + print("Model inputs count:", len(model_inputs)) print("Model input:", model_input) - + print("Model outputs count:", len(model_outputs)) print("Model outputs:") for output in model_outputs: @@ -326,7 +326,7 @@ Load and save an image: .. code:: ipython3 image_path = Path("./data/coco_bike.jpg") - + download_file( url="https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg", filename=image_path.name, @@ -343,7 +343,7 @@ Load and save an image: .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_bike.jpg') @@ -353,16 +353,16 @@ Read the image, resize and convert it to the input shape of the network: # Read the image image = cv2.imread(filename=str(image_path)) - + # The network expects images in RGB format image = cv2.cvtColor(image, code=cv2.COLOR_BGR2RGB) - + # Resize the image to the network input shape resized_image = cv2.resize(src=image, dsize=(255, 255)) - + # Transpose the image to the network input shape network_input_image = np.expand_dims(resized_image, 0) - + # Show the image plt.imshow(image) @@ -371,7 +371,7 @@ Read the image, resize and convert it to the input shape of the network: .. parsed-literal:: - + @@ -396,19 +396,19 @@ outputs will be used. .. code:: ipython3 _, detection_boxes, detection_classes, _, detection_scores, num_detections, _, _ = model_outputs - + image_detection_boxes = inference_result[detection_boxes] print("image_detection_boxes:", image_detection_boxes) - + image_detection_classes = inference_result[detection_classes] print("image_detection_classes:", image_detection_classes) - + image_detection_scores = inference_result[detection_scores] print("image_detection_scores:", image_detection_scores) - + image_num_detections = inference_result[num_detections] print("image_detections_num:", image_num_detections) - + # Alternatively, inference result data can be extracted by model output name with `.get()` method assert (inference_result[detection_boxes] == inference_result.get("detection_boxes")).all(), "extracted inference result data should be equal" @@ -503,12 +503,12 @@ Define utility functions to visualize the inference results import random from typing import Optional - - + + def add_detection_box(box: np.ndarray, image: np.ndarray, label: Optional[str] = None) -> np.ndarray: """ Helper function for adding single bounding box to the image - + Parameters ---------- box : np.ndarray @@ -517,20 +517,20 @@ Define utility functions to visualize the inference results The image to which detection box is added label : str, optional Detection box label string, if not provided will not be added to result image (default is None) - + Returns ------- np.ndarray NumPy array including both image and detection box - + """ ymin, xmin, ymax, xmax = box point1, point2 = (int(xmin), int(ymin)), (int(xmax), int(ymax)) box_color = [random.randint(0, 255) for _ in range(3)] line_thickness = round(0.002 * (image.shape[0] + image.shape[1]) / 2) + 1 - + cv2.rectangle(img=image, pt1=point1, pt2=point2, color=box_color, thickness=line_thickness, lineType=cv2.LINE_AA) - + if label: font_thickness = max(line_thickness - 1, 1) font_face = 0 @@ -551,14 +551,14 @@ Define utility functions to visualize the inference results .. code:: ipython3 from typing import Dict - + from openvino.runtime.utils.data_helpers import OVDict - - + + def visualize_inference_result(inference_result: OVDict, image: np.ndarray, labels_map: Dict, detections_limit: Optional[int] = None): """ Helper function for visualizing inference result on the image - + Parameters ---------- inference_result : OVDict @@ -574,13 +574,13 @@ Define utility functions to visualize the inference results detection_classes: np.ndarray = inference_result.get("detection_classes") detection_scores: np.ndarray = inference_result.get("detection_scores") num_detections: np.ndarray = inference_result.get("num_detections") - + detections_limit = int( min(detections_limit, num_detections[0]) if detections_limit is not None else num_detections[0] ) - + # Normalize detection boxes coordinates to original image size original_image_height, original_image_width, _ = image.shape normalized_detection_boxex = detection_boxes[::] * [ @@ -589,9 +589,9 @@ Define utility functions to visualize the inference results original_image_height, original_image_width, ] - + image_with_detection_boxex = np.copy(image) - + for i in range(detections_limit): detected_class_name = labels_map[int(detection_classes[0, i])] score = detection_scores[0, i] @@ -601,7 +601,7 @@ Define utility functions to visualize the inference results image=image_with_detection_boxex, label=label, ) - + plt.imshow(image_with_detection_boxex) TensorFlow Object Detection model @@ -617,7 +617,7 @@ Zoo `__: .. code:: ipython3 coco_labels_file_path = Path("./data/coco_91cl.txt") - + download_file( url="https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/data/dataset_classes/coco_91cl.txt", filename=coco_labels_file_path.name, @@ -635,7 +635,7 @@ Zoo `__: .. parsed-literal:: - PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt') + PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/120-tensorflow-object-detection-to-openvino/data/coco_91cl.txt') @@ -648,7 +648,7 @@ file: with open(coco_labels_file_path, "r") as file: coco_labels = file.read().strip().split("\n") coco_labels_map = dict(enumerate(coco_labels, 1)) - + print(coco_labels_map) diff --git a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png index c767e2ad495..957606ce732 100644 --- a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png +++ b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/120-tensorflow-object-detection-to-openvino-with-output_38_0.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec79c39b72b7a139eca9f5b7081efec33a47fa8c55c9a459340774a045f2525b -size 391797 +oid sha256:a5b7ad5bc51c70dfab0037ca885aeb4e613947df3fd7fdff8d055c8d64aefd4a +size 391815 diff --git a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html.tmp similarity index 62% rename from docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html rename to docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html.tmp index e489a487635..915c42f4418 100644 --- a/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html +++ b/docs/notebooks/120-tensorflow-object-detection-to-openvino-with-output_files/index.html.tmp @@ -1,8 +1,8 @@ -Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/ +Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/ -

Index of /projects/ov-notebook/0.1.0-latest/20231206220809/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/


../
-120-tensorflow-object-detection-to-openvino-wit..> 07-Dec-2023 00:49              395346
-120-tensorflow-object-detection-to-openvino-wit..> 07-Dec-2023 00:49              391797
+

Index of /projects/ov-notebook/0.1.0-latest/20240125220808/dist/rst_files/120-tensorflow-object-detection-to-openvino-with-output_files/


../
+120-tensorflow-object-detection-to-openvino-wit..> 26-Jan-2024 01:05              395346
+120-tensorflow-object-detection-to-openvino-wit..> 26-Jan-2024 01:05              391815
 

diff --git a/docs/notebooks/121-convert-to-openvino-with-output.rst b/docs/notebooks/121-convert-to-openvino-with-output.rst index 993df191836..4e74150ded1 100644 --- a/docs/notebooks/121-convert-to-openvino-with-output.rst +++ b/docs/notebooks/121-convert-to-openvino-with-output.rst @@ -1,47 +1,50 @@ -OpenVINO™ model conversion API -============================== +OpenVINO™ Model conversion +========================== This notebook shows how to convert a model from original framework format to OpenVINO Intermediate Representation (IR). -**Table of contents:** - +Table of contents: +^^^^^^^^^^^^^^^^^^ - `OpenVINO IR format <#openvino-ir-format>`__ -- `IR preparation with Python conversion API and Model Optimizer - command-line - tool <#ir-preparation-with-python-conversion-api-and-model-optimizer-command-line-tool>`__ - `Fetching example models <#fetching-example-models>`__ -- `Basic conversion <#basic-conversion>`__ -- `Model conversion parameters <#model-conversion-parameters>`__ +- `Conversion <#conversion>`__ - `Setting Input Shapes <#setting-input-shapes>`__ + - `Compressing a Model to FP16 <#compressing-a-model-to-fp16>`__ + - `Convert Models from memory <#convert-models-from-memory>`__ + +- `Migration from Legacy conversion + API <#migration-from-legacy-conversion-api>`__ + + - `Specifying Layout <#specifying-layout>`__ + - `Changing Model Layout <#changing-model-layout>`__ + - `Specifying Mean and Scale + Values <#specifying-mean-and-scale-values>`__ + - `Reversing Input Channels <#reversing-input-channels>`__ - `Cutting Off Parts of a Model <#cutting-off-parts-of-a-model>`__ - - `Embedding Preprocessing - Computation <#embedding-preprocessing-computation>`__ - - - `Specifying Layout <#specifying-layout>`__ - - `Changing Model Layout <#changing-model-layout>`__ - - `Specifying Mean and Scale - Values <#specifying-mean-and-scale-values>`__ - - `Reversing Input Channels <#reversing-input-channels>`__ - - - `Compressing a Model to FP16 <#compressing-a-model-to-fp>`__ - -- `Convert Models Represented as Python - Objects <#convert-models-represented-as-python-objects>`__ .. code:: ipython3 # Required imports. Please execute this cell first. + %pip install --upgrade pip %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu \ - "openvino-dev>=2023.1.0" "requests" "tqdm" "transformers[onnx]>=4.21.1" "torch" "torchvision" + "openvino-dev>=2023.2.0" "requests" "tqdm" "transformers[onnx]>=4.21.1" "torch" "torchvision" "tensorflow_hub" "tensorflow" + + +.. parsed-literal:: + + Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (23.3.2) + + +.. parsed-literal:: + + Note: you may need to restart the kernel to use updated packages. .. parsed-literal:: - ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. - tensorflow 2.12.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 3.20.2 which is incompatible. Note: you may need to restart the kernel to use updated packages. @@ -60,626 +63,78 @@ and biases from the trained model. The resulting IR contains two files: an ``.xml`` file, containing information about network topology, and a ``.bin`` file, containing the weights and biases binary data. -IR preparation with Python conversion API and Model Optimizer command-line tool -------------------------------------------------------------------------------- - - - There are two ways to convert a model from the original framework format -to OpenVINO IR: Python conversion API and Model Optimizer command-line -tool. You can choose one of them based on whichever is most convenient -for you. There should not be any differences in the results of model -conversion if the same set of parameters is used. For more details, -refer to `Model +to OpenVINO IR: Python conversion API and OVC command-line tool. You can +choose one of them based on whichever is most convenient for you. + +OpenVINO conversion API supports next model formats: ``PyTorch``, +``TensorFlow``, ``TensorFlow Lite``, ``ONNX``, and ``PaddlePaddle``. +These model formats can be read, compiled, and converted to OpenVINO IR, +either automatically or explicitly. + +For more details, refer to `Model Preparation `__ documentation. .. code:: ipython3 - # Model Optimizer CLI tool parameters description + # OVC CLI tool parameters description - ! mo --help + ! ovc --help .. parsed-literal:: - usage: main.py [options] + usage: ovc INPUT_MODEL... [-h] [--output_model OUTPUT_MODEL] + [--compress_to_fp16 [True | False]] [--version] [--input INPUT] + [--output OUTPUT] [--extension EXTENSION] [--verbose] + + positional arguments: + INPUT_MODEL Input model file(s) from TensorFlow, ONNX, + PaddlePaddle. Use openvino.convert_model in Python to + convert models from PyTorch. optional arguments: -h, --help show this help message and exit - --framework FRAMEWORK - Name of the framework used to train the input model. - - Framework-agnostic parameters: - --model_name MODEL_NAME, -n MODEL_NAME - Model_name parameter passed to the final create_ir - transform. This parameter is used to name a network in - a generated IR and output .xml/.bin files. - --output_dir OUTPUT_DIR, -o OUTPUT_DIR - Directory that stores the generated IR. By default, it - is the directory from where the Model Conversion is - launched. - --freeze_placeholder_with_value FREEZE_PLACEHOLDER_WITH_VALUE - Replaces input layer with constant node with provided - value, for example: "node_name->True". It will be - DEPRECATED in future releases. Use "input" option to - specify a value for freezing. - --static_shape Enables IR generation for fixed input shape (folding - `ShapeOf` operations and shape-calculating sub-graphs - to `Constant`). Changing model input shape using the - OpenVINO Runtime API in runtime may fail for such an - IR. - --use_new_frontend Force the usage of new Frontend for model conversion - into IR. The new Frontend is C++ based and is - available for ONNX* and PaddlePaddle* models. Model - Conversion API uses new Frontend for ONNX* and - PaddlePaddle* by default that means `use_new_frontend` - and `use_legacy_frontend` options are not specified. - --use_legacy_frontend - Force the usage of legacy Frontend for model - conversion into IR. The legacy Frontend is Python - based and is available for TensorFlow*, ONNX*, MXNet*, - Caffe*, and Kaldi* models. - --input_model INPUT_MODEL, -m INPUT_MODEL, -w INPUT_MODEL - Tensorflow*: a file with a pre-trained model (binary - or text .pb file after freezing). Caffe*: a model - proto file with model weights. - --input INPUT Quoted list of comma-separated input nodes names with - shapes, data types, and values for freezing. The order - of inputs in converted model is the same as order of - specified operation names. The shape and value are - specified as comma-separated lists. The data type of - input node is specified in braces and can have one of - the values: f64 (float64), f32 (float32), f16 - (float16), i64 (int64), i32 (int32), u8 (uint8), - boolean (bool). Data type is optional. If it's not - specified explicitly then there are two options: if - input node is a parameter, data type is taken from the - original node dtype, if input node is not a parameter, - data type is set to f32. Example, to set `input_1` - with shape [1,100], and Parameter node `sequence_len` - with scalar input with value `150`, and boolean input - `is_training` with `False` value use the following - format: - "input_1[1,100],sequence_len->150,is_training->False". - Another example, use the following format to set input - port 0 of the node `node_name1` with the shape [3,4] - as an input node and freeze output port 1 of the node - "node_name2" with the value [20,15] of the int32 type - and shape [2]: - "0:node_name1[3,4],node_name2:1[2]{i32}->[20,15]". - --output OUTPUT The name of the output operation of the model or list - of names. For TensorFlow*, do not add :0 to this - name.The order of outputs in converted model is the - same as order of specified operation names. - --input_shape INPUT_SHAPE - Input shape(s) that should be fed to an input node(s) - of the model. Shape is defined as a comma-separated - list of integer numbers enclosed in parentheses or - square brackets, for example [1,3,227,227] or - (1,227,227,3), where the order of dimensions depends - on the framework input layout of the model. For - example, [N,C,H,W] is used for ONNX* models and - [N,H,W,C] for TensorFlow* models. The shape can - contain undefined dimensions (? or -1) and should fit - the dimensions defined in the input operation of the - graph. Boundaries of undefined dimension can be - specified with ellipsis, for example - [1,1..10,128,128]. One boundary can be undefined, for - example [1,..100] or [1,3,1..,1..]. If there are - multiple inputs in the model, --input_shape should - contain definition of shape for each input separated - by a comma, for example: [1,3,227,227],[2,4] for a - model with two inputs with 4D and 2D shapes. - Alternatively, specify shapes with the --input option. - --example_input EXAMPLE_INPUT - Sample of model input in original framework. For - PyTorch it can be torch.Tensor. For Tensorflow it can - be tf.Tensor or numpy.ndarray. For PaddlePaddle it can - be Paddle Variable. - --batch BATCH, -b BATCH - Set batch size. It applies to 1D or higher dimension - inputs. The default dimension index for the batch is - zero. Use a label 'n' in --layout or --source_layout - option to set the batch dimension. For example, - "x(hwnc)" defines the third dimension to be the batch. - --mean_values MEAN_VALUES - Mean values to be used for the input image per - channel. Values to be provided in the (R,G,B) or - [R,G,B] format. Can be defined for desired input of - the model, for example: "--mean_values - data[255,255,255],info[255,255,255]". The exact - meaning and order of channels depend on how the - original model was trained. - --scale_values SCALE_VALUES - Scale values to be used for the input image per - channel. Values are provided in the (R,G,B) or [R,G,B] - format. Can be defined for desired input of the model, - for example: "--scale_values - data[255,255,255],info[255,255,255]". The exact - meaning and order of channels depend on how the - original model was trained. If both --mean_values and - --scale_values are specified, the mean is subtracted - first and then scale is applied regardless of the - order of options in command line. - --scale SCALE, -s SCALE - All input values coming from original network inputs - will be divided by this value. When a list of inputs - is overridden by the --input parameter, this scale is - not applied for any input that does not match with the - original input of the model. If both --mean_values and - --scale are specified, the mean is subtracted first - and then scale is applied regardless of the order of - options in command line. - --reverse_input_channels [REVERSE_INPUT_CHANNELS] - Switch the input channels order from RGB to BGR (or - vice versa). Applied to original inputs of the model - if and only if a number of channels equals 3. When - --mean_values/--scale_values are also specified, - reversing of channels will be applied to user's input - data first, so that numbers in --mean_values and - --scale_values go in the order of channels used in the - original model. In other words, if both options are - specified, then the data flow in the model looks as - following: Parameter -> ReverseInputChannels -> Mean - apply-> Scale apply -> the original body of the model. - --source_layout SOURCE_LAYOUT - Layout of the input or output of the model in the - framework. Layout can be specified in the short form, - e.g. nhwc, or in complex form, e.g. "[n,h,w,c]". - Example for many names: "in_name1([n,h,w,c]),in_name2( - nc),out_name1(n),out_name2(nc)". Layout can be - partially defined, "?" can be used to specify - undefined layout for one dimension, "..." can be used - to specify undefined layout for multiple dimensions, - for example "?c??", "nc...", "n...c", etc. - --target_layout TARGET_LAYOUT - Same as --source_layout, but specifies target layout - that will be in the model after processing by - ModelOptimizer. - --layout LAYOUT Combination of --source_layout and --target_layout. - Can't be used with either of them. If model has one - input it is sufficient to specify layout of this - input, for example --layout nhwc. To specify layouts - of many tensors, names must be provided, for example: - --layout "name1(nchw),name2(nc)". It is possible to - instruct ModelOptimizer to change layout, for example: - --layout "name1(nhwc->nchw),name2(cn->nc)". Also "*" - in long layout form can be used to fuse dimensions, - for example "[n,c,...]->[n*c,...]". - --compress_to_fp16 [COMPRESS_TO_FP16] - If the original model has FP32 weights or biases, they - are compressed to FP16. All intermediate data is kept - in original precision. Option can be specified alone - as "--compress_to_fp16", or explicit True/False values - can be set, for example: "--compress_to_fp16=False", - or "--compress_to_fp16=True" - --extensions EXTENSIONS + --output_model OUTPUT_MODEL + This parameter is used to name output .xml/.bin files + of converted model. Model name or output directory can + be passed. If output directory is passed, the + resulting .xml/.bin files are named by original model + name. + --compress_to_fp16 [True | False] + Compress weights in output OpenVINO model to FP16. To + turn off compression use "--compress_to_fp16=False" + command line parameter. Default value is True. + --version Print ovc version and exit. + --input INPUT Information of model input required for model + conversion. This is a comma separated list with + optional input names and shapes. The order of inputs + in converted model will match the order of specified + inputs. The shape is specified as comma-separated + list. Example, to set `input_1` input with shape + [1,100] and `sequence_len` input with shape [1,?]: + "input_1[1,100],sequence_len[1,?]", where "?" is a + dynamic dimension, which means that such a dimension + can be specified later in the runtime. If the + dimension is set as an integer (like 100 in [1,100]), + such a dimension is not supposed to be changed later, + during a model conversion it is treated as a static + value. Example with unnamed inputs: "[1,100],[1,?]". + --output OUTPUT One or more comma-separated model outputs to be + preserved in the converted model. Other outputs are + removed. If `output` parameter is not specified then + all outputs from the original model are preserved. Do + not add :0 to the names for TensorFlow. The order of + outputs in the converted model is the same as the + order of specified names. Example: ovc model.onnx + output=out_1,out_2 + --extension EXTENSION Paths or a comma-separated list of paths to libraries - (.so or .dll) with extensions. For the legacy MO path - (if `--use_legacy_frontend` is used), a directory or a - comma-separated list of directories with extensions - are supported. To disable all extensions including - those that are placed at the default location, pass an - empty string. - --transform TRANSFORM - Apply additional transformations. Usage: "--transform - transformation_name1[args],transformation_name2..." - where [args] is key=value pairs separated by - semicolon. Examples: "--transform LowLatency2" or "-- - transform Pruning" or "--transform - LowLatency2[use_const_initializer=False]" or "-- - transform "MakeStateful[param_res_names= {'input_name_ - 1':'output_name_1','input_name_2':'output_name_2'}]" - Available transformations: "LowLatency2", - "MakeStateful", "Pruning" - --transformations_config TRANSFORMATIONS_CONFIG - Use the configuration file with transformations - description. Transformations file can be specified as - relative path from the current directory, as absolute - path or as arelative path from the mo root directory. - --silent [SILENT] Prevent any output messages except those that - correspond to log level equals ERROR, that can be set - with the following option: --log_level. By default, - log level is already ERROR. - --log_level {CRITICAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET} - Logger level of logging massages from MO. Expected one - of ['CRITICAL', 'ERROR', 'WARN', 'WARNING', 'INFO', - 'DEBUG', 'NOTSET']. - --version Version of Model Optimizer - --progress [PROGRESS] - Enable model conversion progress display. - --stream_output [STREAM_OUTPUT] - Switch model conversion progress display to a - multiline mode. - --share_weights [SHARE_WEIGHTS] - Map memory of weights instead reading files or share - memory from input model. Currently, mapping feature is - provided only for ONNX models that do not require - fallback to the legacy ONNX frontend for the - conversion. - - TensorFlow*-specific parameters: - --input_model_is_text [INPUT_MODEL_IS_TEXT] - TensorFlow*: treat the input model file as a text - protobuf format. If not specified, the Model Optimizer - treats it as a binary file by default. - --input_checkpoint INPUT_CHECKPOINT - TensorFlow*: variables file to load. - --input_meta_graph INPUT_META_GRAPH - Tensorflow*: a file with a meta-graph of the model - before freezing - --saved_model_dir SAVED_MODEL_DIR - TensorFlow*: directory with a model in SavedModel - format of TensorFlow 1.x or 2.x version. - --saved_model_tags SAVED_MODEL_TAGS - Group of tag(s) of the MetaGraphDef to load, in string - format, separated by ','. For tag-set contains - multiple tags, all tags must be passed in. - --tensorflow_custom_operations_config_update TENSORFLOW_CUSTOM_OPERATIONS_CONFIG_UPDATE - TensorFlow*: update the configuration file with node - name patterns with input/output nodes information. - --tensorflow_object_detection_api_pipeline_config TENSORFLOW_OBJECT_DETECTION_API_PIPELINE_CONFIG - TensorFlow*: path to the pipeline configuration file - used to generate model created with help of Object - Detection API. - --tensorboard_logdir TENSORBOARD_LOGDIR - TensorFlow*: dump the input graph to a given directory - that should be used with TensorBoard. - --tensorflow_custom_layer_libraries TENSORFLOW_CUSTOM_LAYER_LIBRARIES - TensorFlow*: comma separated list of shared libraries - with TensorFlow* custom operations implementation. - - Caffe*-specific parameters: - --input_proto INPUT_PROTO, -d INPUT_PROTO - Deploy-ready prototxt file that contains a topology - structure and layer attributes - --caffe_parser_path CAFFE_PARSER_PATH - Path to Python Caffe* parser generated from - caffe.proto - --k K Path to CustomLayersMapping.xml to register custom - layers - --disable_omitting_optional [DISABLE_OMITTING_OPTIONAL] - Disable omitting optional attributes to be used for - custom layers. Use this option if you want to transfer - all attributes of a custom layer to IR. Default - behavior is to transfer the attributes with default - values and the attributes defined by the user to IR. - --enable_flattening_nested_params [ENABLE_FLATTENING_NESTED_PARAMS] - Enable flattening optional params to be used for - custom layers. Use this option if you want to transfer - attributes of a custom layer to IR with flattened - nested parameters. Default behavior is to transfer the - attributes without flattening nested parameters. - - MXNet-specific parameters: - --input_symbol INPUT_SYMBOL - Symbol file (for example, model-symbol.json) that - contains a topology structure and layer attributes - --nd_prefix_name ND_PREFIX_NAME - Prefix name for args.nd and argx.nd files. - --pretrained_model_name PRETRAINED_MODEL_NAME - Name of a pretrained MXNet model without extension and - epoch number. This model will be merged with args.nd - and argx.nd files - --save_params_from_nd [SAVE_PARAMS_FROM_ND] - Enable saving built parameters file from .nd files - --legacy_mxnet_model [LEGACY_MXNET_MODEL] - Enable MXNet loader to make a model compatible with - the latest MXNet version. Use only if your model was - trained with MXNet version lower than 1.0.0 - --enable_ssd_gluoncv [ENABLE_SSD_GLUONCV] - Enable pattern matchers replacers for converting - gluoncv ssd topologies. - - Kaldi-specific parameters: - --counts COUNTS Path to the counts file - --remove_output_softmax [REMOVE_OUTPUT_SOFTMAX] - Removes the SoftMax layer that is the output layer - --remove_memory [REMOVE_MEMORY] - Removes the Memory layer and use additional inputs - outputs instead - - -.. code:: ipython3 - - # Python conversion API parameters description - from openvino.tools import mo - - - mo.convert_model(help=True) - - -.. parsed-literal:: - - Optional parameters: - --help - Print available parameters. - --framework - Name of the framework used to train the input model. - - Framework-agnostic parameters: - --input_model - Model object in original framework (PyTorch, Tensorflow) or path to - model file. - Tensorflow*: a file with a pre-trained model (binary or text .pb file - after freezing). - Caffe*: a model proto file with model weights - - Supported formats of input model: - - PaddlePaddle - paddle.hapi.model.Model - paddle.fluid.dygraph.layers.Layer - paddle.fluid.executor.Executor - - PyTorch - torch.nn.Module - torch.jit.ScriptModule - torch.jit.ScriptFunction - - TF - tf.compat.v1.Graph - tf.compat.v1.GraphDef - tf.compat.v1.wrap_function - tf.compat.v1.session - - TF2 / Keras - tf.keras.Model - tf.keras.layers.Layer - tf.function - tf.Module - tf.train.checkpoint - --input - Input can be set by passing a list of InputCutInfo objects or by a list - of tuples. Each tuple can contain optionally input name, input - type or input shape. Example: input=("op_name", PartialShape([-1, - 3, 100, 100]), Type(np.float32)). Alternatively input can be set by - a string or list of strings of the following format. Quoted list of comma-separated - input nodes names with shapes, data types, and values for freezing. - If operation names are specified, the order of inputs in converted - model will be the same as order of specified operation names (applicable - for TF2, ONNX, MxNet). - The shape and value are specified as comma-separated lists. The data - type of input node is specified - in braces and can have one of the values: f64 (float64), f32 (float32), - f16 (float16), i64 - (int64), i32 (int32), u8 (uint8), boolean (bool). Data type is optional. - If it's not specified explicitly then there are two options: if input - node is a parameter, data type is taken from the original node dtype, - if input node is not a parameter, data type is set to f32. Example, to set - `input_1` with shape [1,100], and Parameter node `sequence_len` with - scalar input with value `150`, and boolean input `is_training` with - `False` value use the following format: "input_1[1,100],sequence_len->150,is_training->False". - Another example, use the following format to set input port 0 of the node - `node_name1` with the shape [3,4] as an input node and freeze output - port 1 of the node `node_name2` with the value [20,15] of the int32 type - and shape [2]: "0:node_name1[3,4],node_name2:1[2]{i32}->[20,15]". - - --output - The name of the output operation of the model or list of names. For TensorFlow*, - do not add :0 to this name.The order of outputs in converted model is the - same as order of specified operation names. - --input_shape - Input shape(s) that should be fed to an input node(s) of the model. Input - shapes can be defined by passing a list of objects of type PartialShape, - Shape, [Dimension, ...] or [int, ...] or by a string of the following - format. Shape is defined as a comma-separated list of integer numbers - enclosed in parentheses or square brackets, for example [1,3,227,227] - or (1,227,227,3), where the order of dimensions depends on the framework - input layout of the model. For example, [N,C,H,W] is used for ONNX* models - and [N,H,W,C] for TensorFlow* models. The shape can contain undefined - dimensions (? or -1) and should fit the dimensions defined in the input - operation of the graph. Boundaries of undefined dimension can be specified - with ellipsis, for example [1,1..10,128,128]. One boundary can be - undefined, for example [1,..100] or [1,3,1..,1..]. If there are multiple - inputs in the model, --input_shape should contain definition of shape - for each input separated by a comma, for example: [1,3,227,227],[2,4] - for a model with two inputs with 4D and 2D shapes. Alternatively, specify - shapes with the --input option. - --example_input - Sample of model input in original framework. - For PyTorch it can be torch.Tensor. - For Tensorflow it can be tf.Tensor or numpy.ndarray. - For PaddlePaddle it can be Paddle Variable. - --batch - Set batch size. It applies to 1D or higher dimension inputs. - The default dimension index for the batch is zero. - Use a label 'n' in --layout or --source_layout option to set the batch - dimension. - For example, "x(hwnc)" defines the third dimension to be the batch. - - --mean_values - Mean values to be used for the input image per channel. Mean values can - be set by passing a dictionary, where key is input name and value is mean - value. For example mean_values={'data':[255,255,255],'info':[255,255,255]}. - Or mean values can be set by a string of the following format. Values to - be provided in the (R,G,B) or [R,G,B] format. Can be defined for desired - input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". - The exact meaning and order of channels depend on how the original model - was trained. - --scale_values - Scale values to be used for the input image per channel. Scale values - can be set by passing a dictionary, where key is input name and value is - scale value. For example scale_values={'data':[255,255,255],'info':[255,255,255]}. - Or scale values can be set by a string of the following format. Values - are provided in the (R,G,B) or [R,G,B] format. Can be defined for desired - input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". - The exact meaning and order of channels depend on how the original model - was trained. If both --mean_values and --scale_values are specified, - the mean is subtracted first and then scale is applied regardless of - the order of options in command line. - --scale - All input values coming from original network inputs will be divided - by this value. When a list of inputs is overridden by the --input parameter, - this scale is not applied for any input that does not match with the original - input of the model. If both --mean_values and --scale are specified, - the mean is subtracted first and then scale is applied regardless of - the order of options in command line. - --reverse_input_channels - Switch the input channels order from RGB to BGR (or vice versa). Applied - to original inputs of the model if and only if a number of channels equals - 3. When --mean_values/--scale_values are also specified, reversing - of channels will be applied to user's input data first, so that numbers - in --mean_values and --scale_values go in the order of channels used - in the original model. In other words, if both options are specified, - then the data flow in the model looks as following: Parameter -> ReverseInputChannels - -> Mean apply-> Scale apply -> the original body of the model. - --source_layout - Layout of the input or output of the model in the framework. Layout can - be set by passing a dictionary, where key is input name and value is LayoutMap - object. Or layout can be set by string of the following format. Layout - can be specified in the short form, e.g. nhwc, or in complex form, e.g. - "[n,h,w,c]". Example for many names: "in_name1([n,h,w,c]),in_name2(nc),out_name1(n),out_name2(nc)". - Layout can be partially defined, "?" can be used to specify undefined - layout for one dimension, "..." can be used to specify undefined layout - for multiple dimensions, for example "?c??", "nc...", "n...c", etc. - - --target_layout - Same as --source_layout, but specifies target layout that will be in - the model after processing by ModelOptimizer. - --layout - Combination of --source_layout and --target_layout. Can't be used - with either of them. If model has one input it is sufficient to specify - layout of this input, for example --layout nhwc. To specify layouts - of many tensors, names must be provided, for example: --layout "name1(nchw),name2(nc)". - It is possible to instruct ModelOptimizer to change layout, for example: - --layout "name1(nhwc->nchw),name2(cn->nc)". - Also "*" in long layout form can be used to fuse dimensions, for example - "[n,c,...]->[n*c,...]". - --compress_to_fp16 - If the original model has FP32 weights or biases, they are compressed - to FP16. All intermediate data is kept in original precision. Option - can be specified alone as "--compress_to_fp16", or explicit True/False - values can be set, for example: "--compress_to_fp16=False", or "--compress_to_fp16=True" - - --extensions - Paths to libraries (.so or .dll) with extensions, comma-separated - list of paths, objects derived from BaseExtension class or lists of - objects. For the legacy MO path (if `--use_legacy_frontend` is used), - a directory or a comma-separated list of directories with extensions - are supported. To disable all extensions including those that are placed - at the default location, pass an empty string. - --transform - Apply additional transformations. 'transform' can be set by a list - of tuples, where the first element is transform name and the second element - is transform parameters. For example: [('LowLatency2', {{'use_const_initializer': - False}}), ...]"--transform transformation_name1[args],transformation_name2..." - where [args] is key=value pairs separated by semicolon. Examples: - "--transform LowLatency2" or - "--transform Pruning" or - "--transform LowLatency2[use_const_initializer=False]" or - "--transform "MakeStateful[param_res_names= - {'input_name_1':'output_name_1','input_name_2':'output_name_2'}]"" - Available transformations: "LowLatency2", "MakeStateful", "Pruning" - - --transformations_config - Use the configuration file with transformations description or pass - object derived from BaseExtension class. Transformations file can - be specified as relative path from the current directory, as absolute - path or as relative path from the mo root directory. - --silent - Prevent any output messages except those that correspond to log level - equals ERROR, that can be set with the following option: --log_level. - By default, log level is already ERROR. - --log_level - Logger level of logging massages from MO. - Expected one of ['CRITICAL', 'ERROR', 'WARN', 'WARNING', 'INFO', - 'DEBUG', 'NOTSET']. - --version - Version of Model Optimizer - --progress - Enable model conversion progress display. - --stream_output - Switch model conversion progress display to a multiline mode. - --share_weights - Map memory of weights instead reading files or share memory from input - model. - Currently, mapping feature is provided only for ONNX models - that do not require fallback to the legacy ONNX frontend for the conversion. - - - PaddlePaddle-specific parameters: - --example_output - Sample of model output in original framework. For PaddlePaddle it can - be Paddle Variable. - - TensorFlow*-specific parameters: - --input_model_is_text - TensorFlow*: treat the input model file as a text protobuf format. If - not specified, the Model Optimizer treats it as a binary file by default. - - --input_checkpoint - TensorFlow*: variables file to load. - --input_meta_graph - Tensorflow*: a file with a meta-graph of the model before freezing - --saved_model_dir - TensorFlow*: directory with a model in SavedModel format of TensorFlow - 1.x or 2.x version. - --saved_model_tags - Group of tag(s) of the MetaGraphDef to load, in string format, separated - by ','. For tag-set contains multiple tags, all tags must be passed in. - - --tensorflow_custom_operations_config_update - TensorFlow*: update the configuration file with node name patterns - with input/output nodes information. - --tensorflow_object_detection_api_pipeline_config - TensorFlow*: path to the pipeline configuration file used to generate - model created with help of Object Detection API. - --tensorboard_logdir - TensorFlow*: dump the input graph to a given directory that should be - used with TensorBoard. - --tensorflow_custom_layer_libraries - TensorFlow*: comma separated list of shared libraries with TensorFlow* - custom operations implementation. - - MXNet-specific parameters: - --input_symbol - Symbol file (for example, model-symbol.json) that contains a topology - structure and layer attributes - --nd_prefix_name - Prefix name for args.nd and argx.nd files. - --pretrained_model_name - Name of a pretrained MXNet model without extension and epoch number. - This model will be merged with args.nd and argx.nd files - --save_params_from_nd - Enable saving built parameters file from .nd files - --legacy_mxnet_model - Enable MXNet loader to make a model compatible with the latest MXNet - version. Use only if your model was trained with MXNet version lower - than 1.0.0 - --enable_ssd_gluoncv - Enable pattern matchers replacers for converting gluoncv ssd topologies. - - - Caffe*-specific parameters: - --input_proto - Deploy-ready prototxt file that contains a topology structure and - layer attributes - --caffe_parser_path - Path to Python Caffe* parser generated from caffe.proto - --k - Path to CustomLayersMapping.xml to register custom layers - --disable_omitting_optional - Disable omitting optional attributes to be used for custom layers. - Use this option if you want to transfer all attributes of a custom layer - to IR. Default behavior is to transfer the attributes with default values - and the attributes defined by the user to IR. - --enable_flattening_nested_params - Enable flattening optional params to be used for custom layers. Use - this option if you want to transfer attributes of a custom layer to IR - with flattened nested parameters. Default behavior is to transfer - the attributes without flattening nested parameters. - - Kaldi-specific parameters: - --counts - Path to the counts file - --remove_output_softmax - Removes the SoftMax layer that is the output layer - --remove_memory - Removes the Memory layer and use additional inputs outputs instead - - + (.so or .dll) with extensions. To disable all + extensions including those that are placed at the + default location, pass an empty string. + --verbose Print detailed information about conversion. Fetching example models @@ -711,7 +166,6 @@ NLP model from Hugging Face and export it in ONNX format: from transformers import AutoModelForSequenceClassification, AutoTokenizer from transformers.onnx import export, FeaturesManager - ONNX_NLP_MODEL_PATH = MODEL_DIRECTORY_PATH / "distilbert.onnx" # download model @@ -742,11 +196,19 @@ NLP model from Hugging Face and export it in ONNX format: .. parsed-literal:: - 2023-12-06 23:09:08.345195: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. - 2023-12-06 23:09:08.379671: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + 2024-01-25 23:11:32.267646: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 23:11:32.303561: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. - 2023-12-06 23:09:09.007347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT - /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:223: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. + + +.. parsed-literal:: + + 2024-01-25 23:11:32.956586: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. mask, torch.tensor(torch.finfo(scores.dtype).min) @@ -766,7 +228,6 @@ CV classification model from torchvision: from torchvision.models import resnet50, ResNet50_Weights - # create model object pytorch_model = resnet50(weights=ResNet50_Weights.DEFAULT) # switch model from training to inference mode @@ -963,7 +424,6 @@ Convert PyTorch model to ONNX format: import torch import warnings - ONNX_CV_MODEL_PATH = MODEL_DIRECTORY_PATH / "resnet.onnx" if ONNX_CV_MODEL_PATH.exists(): @@ -972,7 +432,7 @@ Convert PyTorch model to ONNX format: with warnings.catch_warnings(): warnings.filterwarnings("ignore") torch.onnx.export( - model=pytorch_model, args=torch.randn(1, 3, 780, 520), f=ONNX_CV_MODEL_PATH + model=pytorch_model, args=torch.randn(1, 3, 224, 224), f=ONNX_CV_MODEL_PATH ) print(f"ONNX model exported to {ONNX_CV_MODEL_PATH}") @@ -982,51 +442,33 @@ Convert PyTorch model to ONNX format: ONNX model exported to model/resnet.onnx -Basic conversion ----------------- +Conversion +---------- -To convert a model to OpenVINO IR, use the following command: +To convert a model to OpenVINO IR, use the following API: .. code:: ipython3 - # Model Optimizer CLI + import openvino as ov - ! mo --input_model model/distilbert.onnx --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin - - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo - - # mo.convert_model returns an openvino.runtime.Model object - ov_model = mo.convert_model(ONNX_NLP_MODEL_PATH) + # ov.convert_model returns an openvino.runtime.Model object + print(ONNX_NLP_MODEL_PATH) + ov_model = ov.convert_model(ONNX_NLP_MODEL_PATH) # then model can be serialized to *.xml & *.bin files - from openvino.runtime import serialize - - serialize(ov_model, xml_path=MODEL_DIRECTORY_PATH / "distilbert.xml") + ov.save_model(ov_model, MODEL_DIRECTORY_PATH / "distilbert.xml") + + +.. parsed-literal:: + + model/distilbert.onnx + + +.. code:: ipython3 + + ! ovc model/distilbert.onnx --output_model model/distilbert.xml .. parsed-literal:: @@ -1037,36 +479,20 @@ To convert a model to OpenVINO IR, use the following command: - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) -Model conversion parameters ---------------------------- +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False". + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html +.. parsed-literal:: -Both Python conversion API and Model Optimizer command-line tool provide -the following capabilities: \* overriding original input shapes for -model conversion with ``input`` and ``input_shape`` parameters. `Setting -Input Shapes -guide `__. -\* cutting off unwanted parts of a model (such as unsupported operations -and training sub-graphs) using the ``input`` and ``output`` parameters -to define new inputs and outputs of the converted model. `Cutting Off -Parts of a Model -guide `__. -\* inserting additional input pre-processing sub-graphs into the -converted model by using the ``mean_values``, ``scales_values``, -``layout``, and other parameters. `Embedding Preprocessing Computation -article `__. -\* compressing the model weights (for example, weights for convolutions -and matrix multiplications) to FP16 data type using ``compress_to_fp16`` -compression parameter. `Compression of a Model to FP16 -guide `__. + [ SUCCESS ] XML file: model/distilbert.xml + [ SUCCESS ] BIN file: model/distilbert.bin -If the out-of-the-box conversion (only the ``input_model`` parameter is -specified) is not successful, it may be required to use the parameters -mentioned above to override input shapes and cut the model. Setting Input Shapes -~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ @@ -1074,92 +500,24 @@ Model conversion is supported for models with dynamic input shapes that contain undefined dimensions. However, if the shape of data is not going to change from one inference request to another, it is recommended to set up static shapes (when all dimensions are fully defined) for the -inputs. Doing it at this stage, instead of during inference in runtime, -can be beneficial in terms of performance and memory consumption. To set -up static shapes, model conversion API provides the ``input`` and -``input_shape`` parameters. +inputs. Doing so at the model preparation stage, not at runtime, can be +beneficial in terms of performance and memory consumption. -For more information refer to `Setting Input Shapes -guide `__. +For more information refer to `Setting Input +Shapes `__ +documentation. .. code:: ipython3 - # Model Optimizer CLI + import openvino as ov - ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,128],[1,128] --output_dir model - - # alternatively - ! mo --input_model model/distilbert.onnx --input input_ids[1,128],attention_mask[1,128] --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin - - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo - - - ov_model = mo.convert_model( - ONNX_NLP_MODEL_PATH, - input=["input_ids", "attention_mask"], - input_shape=[[1, 128], [1, 128]], - ) - - # alternatively specify input shapes, using the input parameter - ov_model = mo.convert_model( + ov_model = ov.convert_model( ONNX_NLP_MODEL_PATH, input=[("input_ids", [1, 128]), ("attention_mask", [1, 128])] ) -The input_shape parameter allows overriding original input shapes to -ones compatible with a given model. Dynamic shapes, i.e. with dynamic -dimensions, can be replaced in the original model with static shapes for -the converted model, and vice versa. The dynamic dimension can be marked -in the model conversion API parameter as ``-1`` or ``?``. For example, -launch model conversion for the ONNX Bert model and specify a dynamic -sequence length dimension for inputs: - .. code:: ipython3 - # Model Optimizer CLI - - ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,-1],[1,-1] --output_dir model + ! ovc model/distilbert.onnx --input input_ids[1,128],attention_mask[1,128] --output_model model/distilbert.xml .. parsed-literal:: @@ -1172,107 +530,77 @@ sequence length dimension for inputs: .. parsed-literal:: - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False". + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html +.. parsed-literal:: + + [ SUCCESS ] XML file: model/distilbert.xml + [ SUCCESS ] BIN file: model/distilbert.bin + + +The ``input`` parameter allows overriding original input shapes if it is +supported by the model topology. Shapes with dynamic dimensions in the +original model can be replaced with static shapes for the converted +model, and vice versa. The dynamic dimension can be marked in model +conversion API parameter as ``-1`` or ``?`` when using ``ovc``: + .. code:: ipython3 - # Python conversion API - from openvino.tools import mo + import openvino as ov - - ov_model = mo.convert_model( - ONNX_NLP_MODEL_PATH, - input=["input_ids", "attention_mask"], - input_shape=[[1, -1], [1, -1]], + ov_model = ov.convert_model( + ONNX_NLP_MODEL_PATH, input=[("input_ids", [1, -1]), ("attention_mask", [1, -1])] ) +.. code:: ipython3 + + ! ovc model/distilbert.onnx --input "input_ids[1,?],attention_mask[1,?]" --output_model model/distilbert.xml + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False". + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + + +.. parsed-literal:: + + [ SUCCESS ] XML file: model/distilbert.xml + [ SUCCESS ] BIN file: model/distilbert.bin + + To optimize memory consumption for models with undefined dimensions in runtime, model conversion API provides the capability to define -boundaries of dimensions. The boundaries of undefined dimensions can be -specified with ellipsis. For example, launch model conversion for the -ONNX Bert model and specify a boundary for the sequence length -dimension: +boundaries of dimensions. The boundaries of undefined dimension can be +specified with ellipsis in the command line or with +``openvino.Dimension`` class in Python. For example, launch model +conversion for the ONNX Bert model and specify a boundary for the +sequence length dimension: .. code:: ipython3 - # Model Optimizer CLI - - ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,10..128],[1,10..128] --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin - - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo + import openvino as ov - ov_model = mo.convert_model( - ONNX_NLP_MODEL_PATH, - input=["input_ids", "attention_mask"], - input_shape=[[1, "10..128"], [1, "10..128"]], + sequence_length_dim = ov.Dimension(10, 128) + + ov_model = ov.convert_model( + ONNX_NLP_MODEL_PATH, input=[("input_ids", [1, sequence_length_dim]), ("attention_mask", [1, sequence_length_dim])] ) -Cutting Off Parts of a Model -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - - -The following examples show when model cutting is useful or even -required: - -- A model has pre- or post-processing parts that cannot be translated - to existing OpenVINO operations. -- A model has a training part that is convenient to be kept in the - model but not used during inference. -- A model is too complex to be converted at once because it contains - many unsupported operations that cannot be easily implemented as - custom layers. -- A problem occurs with model conversion or inference in OpenVINO - Runtime. To identify the issue, limit the conversion scope by an - iterative search for problematic areas in the model. -- A single custom layer or a combination of custom layers is isolated - for debugging purposes. - -For a more detailed description, refer to the `Cutting Off Parts of a -Model -guide `__. - .. code:: ipython3 - # Model Optimizer CLI - - # cut at the end - ! mo --input_model model/distilbert.onnx --output /classifier/Gemm --output_dir model - - - # cut from the beginning - ! mo --input_model model/distilbert.onnx --input /distilbert/embeddings/LayerNorm/Add_1,attention_mask --output_dir model + ! ovc model/distilbert.onnx --input input_ids[1,10..128],attention_mask[1,10..128] --output_model model/distilbert.xml .. parsed-literal:: @@ -1285,13 +613,36 @@ guide `__, +`TensorFlow `__, +`PaddlePaddle `__ +frameworks conversion guides. + +.. code:: ipython3 + + import openvino as ov + import torch + + example_input = torch.rand(1, 3, 224, 224) + + ov_model = ov.convert_model(pytorch_model, example_input=example_input, input=example_input.shape) + + +.. parsed-literal:: + + WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11. .. code:: ipython3 - # Python conversion API - from openvino.tools import mo + import openvino as ov + import tensorflow_hub as hub + model = hub.load("https://www.kaggle.com/models/google/movenet/frameworks/TensorFlow2/variations/singlepose-lightning/versions/4") + movenet = model.signatures['serving_default'] - # cut at the end - ov_model = mo.convert_model(ONNX_NLP_MODEL_PATH, output="/classifier/Gemm") - - # cut from the beginning - ov_model = mo.convert_model( - ONNX_NLP_MODEL_PATH, - input=["/distilbert/embeddings/LayerNorm/Add_1", "attention_mask"], - ) + ov_model = ov.convert_model(movenet) -Embedding Preprocessing Computation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. parsed-literal:: + + 2024-01-25 23:11:51.061555: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW + 2024-01-25 23:11:51.061587: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07 + 2024-01-25 23:11:51.061592: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07 + 2024-01-25 23:11:51.061773: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2 + 2024-01-25 23:11:51.061789: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3 + 2024-01-25 23:11:51.061793: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration + + +Migration from Legacy conversion API +------------------------------------ -Input data for inference can be different from the training dataset and -requires additional preprocessing before inference. To accelerate the -whole pipeline, including preprocessing and inference, model conversion -API provides special parameters such as ``mean_values``, -``scale_values``, ``reverse_input_channels``, and ``layout``. Based on -these parameters, model conversion API generates OpenVINO IR with -additionally inserted sub-graphs to perform the defined preprocessing. -This preprocessing block can perform mean-scale normalization of input -data, reverting data along channel dimension, and changing the data -layout. For more information on preprocessing, refer to the `Embedding -Preprocessing Computation -article `__. +In the 2023.1 OpenVINO release OpenVINO Model Conversion API was +introduced with the corresponding Python API: ``openvino.convert_model`` +method. ``ovc`` and ``openvino.convert_model`` represent a lightweight +alternative of ``mo`` and ``openvino.tools.mo.convert_model`` which are +considered legacy API now. ``mo.convert_model()`` provides a wide range +of preprocessing parameters. Most of these parameters have analogs in +OVC or can be replaced with functionality from ``ov.PrePostProcessor`` +class. Refer to `Optimize Preprocessing +notebook `__ +for more information about `Preprocessing +API `__. +Here is the migration guide from legacy model preprocessing to +Preprocessing API. Specifying Layout ^^^^^^^^^^^^^^^^^ @@ -1360,14 +740,26 @@ overview nhwc" --output_dir model + ov_model = ov.convert_model(ONNX_CV_MODEL_PATH) - # alternatively use source_layout and target_layout parameters - ! mo --input_model model/resnet.onnx --source_layout nchw --target_layout nhwc --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - + prep = ov.preprocess.PrePostProcessor(ov_model) + prep.input('input.1').tensor().set_layout(ov.Layout("nhwc")) + prep.input('input.1').model().set_layout(ov.Layout("nchw")) + ov_model = prep.build() .. code:: ipython3 - # Python conversion API + # Legacy Model Optimizer API from openvino.tools import mo - ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, layout="nchw->nhwc") # alternatively use source_layout and target_layout parameters @@ -1472,69 +812,37 @@ Specifying Mean and Scale Values -Model conversion API has the following parameters to specify the values: -``mean_values``, ``scale_values``, ``scale``. Using these parameters, -model conversion API embeds the corresponding preprocessing block for -mean-value normalization of the input data and optimizes this block so -that the preprocessing takes negligible time for inference. +Using Preprocessing API ``mean`` and ``scale`` values can be set. Using +these API, model embeds the corresponding preprocessing block for +mean-value normalization of the input data and optimizes this block. +Refer to `Optimize Preprocessing +notebook `__ +for more examples. .. code:: ipython3 - # Model Optimizer CLI + # Converter API + import openvino as ov - ! mo --input_model model/resnet.onnx --mean_values [123,117,104] --scale 255 --output_dir model + ov_model = ov.convert_model(ONNX_CV_MODEL_PATH) - ! mo --input_model model/resnet.onnx --mean_values [123,117,104] --scale_values [255,255,255] --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - + prep = ov.preprocess.PrePostProcessor(ov_model) + prep.input("input.1").tensor().set_layout(ov.Layout("nchw")) + prep.input("input.1").preprocess().mean([255 * x for x in [0.485, 0.456, 0.406]]) + prep.input("input.1").preprocess().scale([255 * x for x in [0.229, 0.224, 0.225]]) + + ov_model = prep.build() .. code:: ipython3 - # Python conversion API + # Legacy Model Optimizer API from openvino.tools import mo - ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, mean_values=[123, 117, 104], scale=255) - ov_model = mo.convert_model( - ONNX_CV_MODEL_PATH, mean_values=[123, 117, 104], scale_values=[255, 255, 255] + ONNX_CV_MODEL_PATH, + mean_values=[255 * x for x in [0.485, 0.456, 0.406]], + scale_values=[255 * x for x in [0.229, 0.224, 0.225]], ) Reversing Input Channels @@ -1550,126 +858,33 @@ the color channels before inference. .. code:: ipython3 - # Model Optimizer CLI + # Converter API + import openvino as ov - ! mo --input_model model/resnet.onnx --reverse_input_channels --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - + ov_model = ov.convert_model(ONNX_CV_MODEL_PATH) + + prep = ov.preprocess.PrePostProcessor(ov_model) + prep.input('input.1').tensor().set_layout(ov.Layout("nchw")) + prep.input('input.1').preprocess().reverse_channels() + ov_model = prep.build() .. code:: ipython3 - # Python conversion API + # Legacy Model Optimizer API from openvino.tools import mo - ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, reverse_input_channels=True) -Compressing a Model to FP16 -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Cutting Off Parts of a Model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Optionally all relevant floating-point weights can be compressed to FP16 -data type during the model conversion, creating a compressed FP16 model. -This smaller model occupies about half of the original space in the file -system. While the compression may introduce a drop in accuracy, for most -models, this decrease is negligible. - -.. code:: ipython3 - - # Model Optimizer CLI - - ! mo --input_model model/resnet.onnx --compress_to_fp16=True --output_dir model - - -.. parsed-literal:: - - huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... - To disable this warning, you can either: - - Avoid using `tokenizers` before the fork if possible - - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) - - -.. parsed-literal:: - - [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. - Find more information about compression to FP16 at https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_FP16_Compression.html - [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. - Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml - [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-561/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin - - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo - - - ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, compress_to_fp16=True) - -Convert Models Represented as Python Objects --------------------------------------------- - - - -Python conversion API can pass Python model objects, such as a Pytorch -model or TensorFlow Keras model directly, without saving them into files -and without leaving the training environment (Jupyter Notebook or -training scripts). - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo - - - ov_model = mo.convert_model(pytorch_model) - - -.. parsed-literal:: - - WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11. - - -``convert_model()`` accepts all parameters available in the MO -command-line tool. Parameters can be specified by Python classes or -string analogs, similar to the command-line tool. - -.. code:: ipython3 - - # Python conversion API - from openvino.tools import mo - - - ov_model = mo.convert_model( - pytorch_model, - input_shape=[1, 3, 100, 100], - mean_values=[127, 127, 127], - layout="nchw", - ) - - ov_model = mo.convert_model(pytorch_model, source_layout="nchw", target_layout="nhwc") - - ov_model = mo.convert_model( - pytorch_model, compress_to_fp16=True, reverse_input_channels=True - ) +Cutting model inputs and outputs from a model is no longer available in +the new conversion API. Instead, we recommend performing the cut in the +original framework. Examples of model cutting of TensorFlow protobuf, +TensorFlow SavedModel, and ONNX formats with tools provided by the +Tensorflow and ONNX frameworks can be found in `documentation +guide `__. +For PyTorch, TensorFlow 2 Keras, and PaddlePaddle, we recommend changing +the original model code to perform the model cut. diff --git a/docs/notebooks/121-legacy-mo-convert-to-openvino-with-output.rst b/docs/notebooks/121-legacy-mo-convert-to-openvino-with-output.rst new file mode 100644 index 00000000000..b13d9b8f26c --- /dev/null +++ b/docs/notebooks/121-legacy-mo-convert-to-openvino-with-output.rst @@ -0,0 +1,1769 @@ +OpenVINO™ model conversion API +============================== + +This notebook shows how to convert a model from original framework +format to OpenVINO Intermediate Representation (IR). + +Table of contents: +^^^^^^^^^^^^^^^^^^ + +- `OpenVINO IR format <#openvino-ir-format>`__ +- `IR preparation with Python conversion API and Model Optimizer + command-line + tool <#ir-preparation-with-python-conversion-api-and-model-optimizer-command-line-tool>`__ +- `Fetching example models <#fetching-example-models>`__ +- `Basic conversion <#basic-conversion>`__ +- `Model conversion parameters <#model-conversion-parameters>`__ + + - `Setting Input Shapes <#setting-input-shapes>`__ + - `Cutting Off Parts of a Model <#cutting-off-parts-of-a-model>`__ + - `Embedding Preprocessing + Computation <#embedding-preprocessing-computation>`__ + + - `Specifying Layout <#specifying-layout>`__ + - `Changing Model Layout <#changing-model-layout>`__ + - `Specifying Mean and Scale + Values <#specifying-mean-and-scale-values>`__ + - `Reversing Input Channels <#reversing-input-channels>`__ + + - `Compressing a Model to FP16 <#compressing-a-model-to-fp16>`__ + +- `Convert Models Represented as Python + Objects <#convert-models-represented-as-python-objects>`__ + +.. code:: ipython3 + + # Required imports. Please execute this cell first. + %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu \ + "openvino-dev>=2023.1.0" "requests" "tqdm" "transformers[onnx]>=4.21.1" "torch" "torchvision" + + +.. parsed-literal:: + + Note: you may need to restart the kernel to use updated packages. + + +OpenVINO IR format +------------------ + + + +OpenVINO `Intermediate Representation +(IR) `__ is the +proprietary model format of OpenVINO. It is produced after converting a +model with model conversion API. Model conversion API translates the +frequently used deep learning operations to their respective similar +representation in OpenVINO and tunes them with the associated weights +and biases from the trained model. The resulting IR contains two files: +an ``.xml`` file, containing information about network topology, and a +``.bin`` file, containing the weights and biases binary data. + +IR preparation with Python conversion API and Model Optimizer command-line tool +------------------------------------------------------------------------------- + + + +There are two ways to convert a model from the original framework format +to OpenVINO IR: Python conversion API and Model Optimizer command-line +tool. You can choose one of them based on whichever is most convenient +for you. There should not be any differences in the results of model +conversion if the same set of parameters is used. For more details, +refer to `Model +Preparation `__ +documentation. + +.. code:: ipython3 + + # Model Optimizer CLI tool parameters description + + ! mo --help + + +.. parsed-literal:: + + usage: main.py [options] + + optional arguments: + -h, --help show this help message and exit + --framework FRAMEWORK + Name of the framework used to train the input model. + + Framework-agnostic parameters: + --model_name MODEL_NAME, -n MODEL_NAME + Model_name parameter passed to the final create_ir + transform. This parameter is used to name a network in + a generated IR and output .xml/.bin files. + --output_dir OUTPUT_DIR, -o OUTPUT_DIR + Directory that stores the generated IR. By default, it + is the directory from where the Model Conversion is + launched. + --freeze_placeholder_with_value FREEZE_PLACEHOLDER_WITH_VALUE + Replaces input layer with constant node with provided + value, for example: "node_name->True". It will be + DEPRECATED in future releases. Use "input" option to + specify a value for freezing. + --static_shape Enables IR generation for fixed input shape (folding + `ShapeOf` operations and shape-calculating sub-graphs + to `Constant`). Changing model input shape using the + OpenVINO Runtime API in runtime may fail for such an + IR. + --use_new_frontend Force the usage of new Frontend for model conversion + into IR. The new Frontend is C++ based and is + available for ONNX* and PaddlePaddle* models. Model + Conversion API uses new Frontend for ONNX* and + PaddlePaddle* by default that means `use_new_frontend` + and `use_legacy_frontend` options are not specified. + --use_legacy_frontend + Force the usage of legacy Frontend for model + conversion into IR. The legacy Frontend is Python + based and is available for TensorFlow*, ONNX*, MXNet*, + Caffe*, and Kaldi* models. + --input_model INPUT_MODEL, -w INPUT_MODEL, -m INPUT_MODEL + Tensorflow*: a file with a pre-trained model (binary + or text .pb file after freezing). Caffe*: a model + proto file with model weights. + --input INPUT Quoted list of comma-separated input nodes names with + shapes, data types, and values for freezing. The order + of inputs in converted model is the same as order of + specified operation names. The shape and value are + specified as comma-separated lists. The data type of + input node is specified in braces and can have one of + the values: f64 (float64), f32 (float32), f16 + (float16), i64 (int64), i32 (int32), u8 (uint8), + boolean (bool). Data type is optional. If it's not + specified explicitly then there are two options: if + input node is a parameter, data type is taken from the + original node dtype, if input node is not a parameter, + data type is set to f32. Example, to set `input_1` + with shape [1,100], and Parameter node `sequence_len` + with scalar input with value `150`, and boolean input + `is_training` with `False` value use the following + format: + "input_1[1,100],sequence_len->150,is_training->False". + Another example, use the following format to set input + port 0 of the node `node_name1` with the shape [3,4] + as an input node and freeze output port 1 of the node + "node_name2" with the value [20,15] of the int32 type + and shape [2]: + "0:node_name1[3,4],node_name2:1[2]{i32}->[20,15]". + --output OUTPUT The name of the output operation of the model or list + of names. For TensorFlow*, do not add :0 to this + name.The order of outputs in converted model is the + same as order of specified operation names. + --input_shape INPUT_SHAPE + Input shape(s) that should be fed to an input node(s) + of the model. Shape is defined as a comma-separated + list of integer numbers enclosed in parentheses or + square brackets, for example [1,3,227,227] or + (1,227,227,3), where the order of dimensions depends + on the framework input layout of the model. For + example, [N,C,H,W] is used for ONNX* models and + [N,H,W,C] for TensorFlow* models. The shape can + contain undefined dimensions (? or -1) and should fit + the dimensions defined in the input operation of the + graph. Boundaries of undefined dimension can be + specified with ellipsis, for example + [1,1..10,128,128]. One boundary can be undefined, for + example [1,..100] or [1,3,1..,1..]. If there are + multiple inputs in the model, --input_shape should + contain definition of shape for each input separated + by a comma, for example: [1,3,227,227],[2,4] for a + model with two inputs with 4D and 2D shapes. + Alternatively, specify shapes with the --input option. + --example_input EXAMPLE_INPUT + Sample of model input in original framework. For + PyTorch it can be torch.Tensor. For Tensorflow it can + be tf.Tensor or numpy.ndarray. For PaddlePaddle it can + be Paddle Variable. + --batch BATCH, -b BATCH + Set batch size. It applies to 1D or higher dimension + inputs. The default dimension index for the batch is + zero. Use a label 'n' in --layout or --source_layout + option to set the batch dimension. For example, + "x(hwnc)" defines the third dimension to be the batch. + --mean_values MEAN_VALUES + Mean values to be used for the input image per + channel. Values to be provided in the (R,G,B) or + [R,G,B] format. Can be defined for desired input of + the model, for example: "--mean_values + data[255,255,255],info[255,255,255]". The exact + meaning and order of channels depend on how the + original model was trained. + --scale_values SCALE_VALUES + Scale values to be used for the input image per + channel. Values are provided in the (R,G,B) or [R,G,B] + format. Can be defined for desired input of the model, + for example: "--scale_values + data[255,255,255],info[255,255,255]". The exact + meaning and order of channels depend on how the + original model was trained. If both --mean_values and + --scale_values are specified, the mean is subtracted + first and then scale is applied regardless of the + order of options in command line. + --scale SCALE, -s SCALE + All input values coming from original network inputs + will be divided by this value. When a list of inputs + is overridden by the --input parameter, this scale is + not applied for any input that does not match with the + original input of the model. If both --mean_values and + --scale are specified, the mean is subtracted first + and then scale is applied regardless of the order of + options in command line. + --reverse_input_channels [REVERSE_INPUT_CHANNELS] + Switch the input channels order from RGB to BGR (or + vice versa). Applied to original inputs of the model + if and only if a number of channels equals 3. When + --mean_values/--scale_values are also specified, + reversing of channels will be applied to user's input + data first, so that numbers in --mean_values and + --scale_values go in the order of channels used in the + original model. In other words, if both options are + specified, then the data flow in the model looks as + following: Parameter -> ReverseInputChannels -> Mean + apply-> Scale apply -> the original body of the model. + --source_layout SOURCE_LAYOUT + Layout of the input or output of the model in the + framework. Layout can be specified in the short form, + e.g. nhwc, or in complex form, e.g. "[n,h,w,c]". + Example for many names: "in_name1([n,h,w,c]),in_name2( + nc),out_name1(n),out_name2(nc)". Layout can be + partially defined, "?" can be used to specify + undefined layout for one dimension, "..." can be used + to specify undefined layout for multiple dimensions, + for example "?c??", "nc...", "n...c", etc. + --target_layout TARGET_LAYOUT + Same as --source_layout, but specifies target layout + that will be in the model after processing by + ModelOptimizer. + --layout LAYOUT Combination of --source_layout and --target_layout. + Can't be used with either of them. If model has one + input it is sufficient to specify layout of this + input, for example --layout nhwc. To specify layouts + of many tensors, names must be provided, for example: + --layout "name1(nchw),name2(nc)". It is possible to + instruct ModelOptimizer to change layout, for example: + --layout "name1(nhwc->nchw),name2(cn->nc)". Also "*" + in long layout form can be used to fuse dimensions, + for example "[n,c,...]->[n*c,...]". + --compress_to_fp16 [COMPRESS_TO_FP16] + If the original model has FP32 weights or biases, they + are compressed to FP16. All intermediate data is kept + in original precision. Option can be specified alone + as "--compress_to_fp16", or explicit True/False values + can be set, for example: "--compress_to_fp16=False", + or "--compress_to_fp16=True" + --extensions EXTENSIONS + Paths or a comma-separated list of paths to libraries + (.so or .dll) with extensions. For the legacy MO path + (if `--use_legacy_frontend` is used), a directory or a + comma + +.. parsed-literal:: + + -separated list of directories with extensions + are supported. To disable all extensions including + those that are placed at the default location, pass an + empty string. + --transform TRANSFORM + Apply additional transformations. Usage: "--transform + transformation_name1[args],transformation_name2..." + where [args] is key=value pairs separated by + semicolon. Examples: "--transform LowLatency2" or "-- + transform Pruning" or "--transform + LowLatency2[use_const_initializer=False]" or "-- + transform "MakeStateful[param_res_names= {'input_name_ + 1':'output_name_1','input_name_2':'output_name_2'}]" + Available transformations: "LowLatency2", + "MakeStateful", "Pruning" + --transformations_config TRANSFORMATIONS_CONFIG + Use the configuration file with transformations + description. Transformations file can be specified as + relative path from the current directory, as absolute + path or as arelative path from the mo root directory. + --silent [SILENT] Prevent any output messages except those that + correspond to log level equals ERROR, that can be set + with the following option: --log_level. By default, + log level is already ERROR. + --log_level {CRITICAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET} + Logger level of logging massages from MO. Expected one + of ['CRITICAL', 'ERROR', 'WARN', 'WARNING', 'INFO', + 'DEBUG', 'NOTSET']. + --version Version of Model Optimizer + --progress [PROGRESS] + Enable model conversion progress display. + --stream_output [STREAM_OUTPUT] + Switch model conversion progress display to a + multiline mode. + --share_weights [SHARE_WEIGHTS] + Map memory of weights instead reading files or share + memory from input model. Currently, mapping feature is + provided only for ONNX models that do not require + fallback to the legacy ONNX frontend for the + conversion. + + TensorFlow*-specific parameters: + --input_model_is_text [INPUT_MODEL_IS_TEXT] + TensorFlow*: treat the input model file as a text + protobuf format. If not specified, the Model Optimizer + treats it as a binary file by default. + --input_checkpoint INPUT_CHECKPOINT + TensorFlow*: variables file to load. + --input_meta_graph INPUT_META_GRAPH + Tensorflow*: a file with a meta-graph of the model + before freezing + --saved_model_dir SAVED_MODEL_DIR + TensorFlow*: directory with a model in SavedModel + format of TensorFlow 1.x or 2.x version. + --saved_model_tags SAVED_MODEL_TAGS + Group of tag(s) of the MetaGraphDef to load, in string + format, separated by ','. For tag-set contains + multiple tags, all tags must be passed in. + --tensorflow_custom_operations_config_update TENSORFLOW_CUSTOM_OPERATIONS_CONFIG_UPDATE + TensorFlow*: update the configuration file with node + name patterns with input/output nodes information. + --tensorflow_object_detection_api_pipeline_config TENSORFLOW_OBJECT_DETECTION_API_PIPELINE_CONFIG + TensorFlow*: path to the pipeline configuration file + used to generate model created with help of Object + Detection API. + --tensorboard_logdir TENSORBOARD_LOGDIR + TensorFlow*: dump the input graph to a given directory + that should be used with TensorBoard. + --tensorflow_custom_layer_libraries TENSORFLOW_CUSTOM_LAYER_LIBRARIES + TensorFlow*: comma separated list of shared libraries + with TensorFlow* custom operations implementation. + + Caffe*-specific parameters: + --input_proto INPUT_PROTO, -d INPUT_PROTO + Deploy-ready prototxt file that contains a topology + structure and layer attributes + --caffe_parser_path CAFFE_PARSER_PATH + Path to Python Caffe* parser generated from + caffe.proto + --k K Path to CustomLayersMapping.xml to register custom + layers + --disable_omitting_optional [DISABLE_OMITTING_OPTIONAL] + Disable omitting optional attributes to be used for + custom layers. Use this option if you want to transfer + all attributes of a custom layer to IR. Default + behavior is to transfer the attributes with default + values and the attributes defined by the user to IR. + --enable_flattening_nested_params [ENABLE_FLATTENING_NESTED_PARAMS] + Enable flattening optional params to be used for + custom layers. Use this option if you want to transfer + attributes of a custom layer to IR with flattened + nested parameters. Default behavior is to transfer the + attributes without flattening nested parameters. + + MXNet-specific parameters: + --input_symbol INPUT_SYMBOL + Symbol file (for example, model-symbol.json) that + contains a topology structure and layer attributes + --nd_prefix_name ND_PREFIX_NAME + Prefix name for args.nd and argx.nd files. + --pretrained_model_name PRETRAINED_MODEL_NAME + Name of a pretrained MXNet model without extension and + epoch number. This model will be merged with args.nd + and argx.nd files + --save_params_from_nd [SAVE_PARAMS_FROM_ND] + Enable saving built parameters file from .nd files + --legacy_mxnet_model [LEGACY_MXNET_MODEL] + Enable MXNet loader to make a model compatible with + the latest MXNet version. Use only if your model was + trained with MXNet version lower than 1.0.0 + --enable_ssd_gluoncv [ENABLE_SSD_GLUONCV] + Enable pattern matchers replacers for converting + gluoncv ssd topologies. + + Kaldi-specific parameters: + --counts COUNTS Path to the counts file + --remove_output_softmax [REMOVE_OUTPUT_SOFTMAX] + Removes the SoftMax layer that is the output layer + --remove_memory [REMOVE_MEMORY] + Removes the Memory layer and use additional inputs + outputs instead + + +.. code:: ipython3 + + # Python conversion API parameters description + from openvino.tools import mo + + + mo.convert_model(help=True) + + +.. parsed-literal:: + + Optional parameters: + --help + Print available parameters. + --framework + Name of the framework used to train the input model. + + Framework-agnostic parameters: + --input_model + Model object in original framework (PyTorch, Tensorflow) or path to + model file. + Tensorflow*: a file with a pre-trained model (binary or text .pb file + after freezing). + Caffe*: a model proto file with model weights + + Supported formats of input model: + + PaddlePaddle + paddle.hapi.model.Model + paddle.fluid.dygraph.layers.Layer + paddle.fluid.executor.Executor + + PyTorch + torch.nn.Module + torch.jit.ScriptModule + torch.jit.ScriptFunction + + TF + tf.compat.v1.Graph + tf.compat.v1.GraphDef + tf.compat.v1.wrap_function + tf.compat.v1.session + + TF2 / Keras + tf.keras.Model + tf.keras.layers.Layer + tf.function + tf.Module + tf.train.checkpoint + --input + Input can be set by passing a list of InputCutInfo objects or by a list + of tuples. Each tuple can contain optionally input name, input + type or input shape. Example: input=("op_name", PartialShape([-1, + 3, 100, 100]), Type(np.float32)). Alternatively input can be set by + a string or list of strings of the following format. Quoted list of comma-separated + input nodes names with shapes, data types, and values for freezing. + If operation names are specified, the order of inputs in converted + model will be the same as order of specified operation names (applicable + for TF2, ONNX, MxNet). + The shape and value are specified as comma-separated lists. The data + type of input node is specified + in braces and can have one of the values: f64 (float64), f32 (float32), + f16 (float16), i64 + (int64), i32 (int32), u8 (uint8), boolean (bool). Data type is optional. + If it's not specified explicitly then there are two options: if input + node is a parameter, data type is taken from the original node dtype, + if input node is not a parameter, data type is set to f32. Example, to set + `input_1` with shape [1,100], and Parameter node `sequence_len` with + scalar input with value `150`, and boolean input `is_training` with + `False` value use the following format: "input_1[1,100],sequence_len->150,is_training->False". + Another example, use the following format to set input port 0 of the node + `node_name1` with the shape [3,4] as an input node and freeze output + port 1 of the node `node_name2` with the value [20,15] of the int32 type + and shape [2]: "0:node_name1[3,4],node_name2:1[2]{i32}->[20,15]". + + --output + The name of the output operation of the model or list of names. For TensorFlow*, + do not add :0 to this name.The order of outputs in converted model is the + same as order of specified operation names. + --input_shape + Input shape(s) that should be fed to an input node(s) of the model. Input + shapes can be defined by passing a list of objects of type PartialShape, + Shape, [Dimension, ...] or [int, ...] or by a string of the following + format. Shape is defined as a comma-separated list of integer numbers + enclosed in parentheses or square brackets, for example [1,3,227,227] + or (1,227,227,3), where the order of dimensions depends on the framework + input layout of the model. For example, [N,C,H,W] is used for ONNX* models + and [N,H,W,C] for TensorFlow* models. The shape can contain undefined + dimensions (? or -1) and should fit the dimensions defined in the input + operation of the graph. Boundaries of undefined dimension can be specified + with ellipsis, for example [1,1..10,128,128]. One boundary can be + undefined, for example [1,..100] or [1,3,1..,1..]. If there are multiple + inputs in the model, --input_shape should contain definition of shape + for each input separated by a comma, for example: [1,3,227,227],[2,4] + for a model with two inputs with 4D and 2D shapes. Alternatively, specify + shapes with the --input option. + --example_input + Sample of model input in original framework. + For PyTorch it can be torch.Tensor. + For Tensorflow it can be tf.Tensor or numpy.ndarray. + For PaddlePaddle it can be Paddle Variable. + --batch + Set batch size. It applies to 1D or higher dimension inputs. + The default dimension index for the batch is zero. + Use a label 'n' in --layout or --source_layout option to set the batch + dimension. + For example, "x(hwnc)" defines the third dimension to be the batch. + + --mean_values + Mean values to be used for the input image per channel. Mean values can + be set by passing a dictionary, where key is input name and value is mean + value. For example mean_values={'data':[255,255,255],'info':[255,255,255]}. + Or mean values can be set by a string of the following format. Values to + be provided in the (R,G,B) or [R,G,B] format. Can be defined for desired + input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". + The exact meaning and order of channels depend on how the original model + was trained. + --scale_values + Scale values to be used for the input image per channel. Scale values + can be set by passing a dictionary, where key is input name and value is + scale value. For example scale_values={'data':[255,255,255],'info':[255,255,255]}. + Or scale values can be set by a string of the following format. Values + are provided in the (R,G,B) or [R,G,B] format. Can be defined for desired + input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". + The exact meaning and order of channels depend on how the original model + was trained. If both --mean_values and --scale_values are specified, + the mean is subtracted first and then scale is applied regardless of + the order of options in command line. + --scale + All input values coming from original network inputs will be divided + by this value. When a list of inputs is overridden by the --input parameter, + this scale is not applied for any input that does not match with the original + input of the model. If both --mean_values and --scale are specified, + the mean is subtracted first and then scale is applied regardless of + the order of options in command line. + --reverse_input_channels + Switch the input channels order from RGB to BGR (or vice versa). Applied + to original inputs of the model if and only if a number of channels equals + 3. When --mean_values/--scale_values are also specified, reversing + of channels will be applied to user's input data first, so that numbers + in --mean_values and --scale_values go in the order of channels used + in the original model. In other words, if both options are specified, + then the data flow in the model looks as following: Parameter -> ReverseInputChannels + -> Mean apply-> Scale apply -> the original body of the model. + --source_layout + Layout of the input or output of the model in the framework. Layout can + be set by passing a dictionary, where key is input name and value is LayoutMap + object. Or layout can be set by string of the following format. Layout + can be specified in the short form, e.g. nhwc, or in complex form, e.g. + "[n,h,w,c]". Example for many names: "in_name1([n,h,w,c]),in_name2(nc),out_name1(n),out_name2(nc)". + Layout can be partially defined, "?" can be used to specify undefined + layout for one dimension, "..." can be used to specify undefined layout + for multiple dimensions, for example "?c??", "nc...", "n...c", etc. + + --target_layout + Same as --source_layout, but specifies target layout that will be in + the model after processing by ModelOptimizer. + --layout + Combination of --source_layout and --target_layout. Can't be used + with either of them. If model has one input it is sufficient to specify + layout of this input, for example --layout nhwc. To specify layouts + of many tensors, names must be provided, for example: --layout "name1(nchw),name2(nc)". + It is possible to instruct ModelOptimizer to change layout, for example: + --layout "name1(nhwc->nchw),name2(cn->nc)". + Also "*" in long layout form can be used to fuse dimensions, for example + "[n,c,...]->[n*c,...]". + --compress_to_fp16 + If the original model has FP32 weights or biases, they are compressed + to FP16. All intermediate data is kept in original precision. Option + can be specified alone as "--compress_to_fp16", or explicit True/False + values can be set, for example: "--compress_to_fp16=False", or "--compress_to_fp16=True" + + --extensions + Paths to libraries (.so or .dll) with extensions, comma-separated + list of paths, objects derived from BaseExtension class or lists of + objects. For the legacy MO path (if `--use_legacy_frontend` is used), + a directory or a comma-separated list of directories with extensions + are supported. To disable all extensions including those that are placed + at the default location, pass an empty string. + --transform + Apply additional transformations. 'transform' can be set by a list + of tuples, where the first element is transform name and the second element + is transform parameters. For example: [('LowLatency2', {{'use_const_initializer': + False}}), ...]"--transform transformation_name1[args],transformation_name2..." + where [args] is key=value pairs separated by semicolon. Examples: + "--transform LowLatency2" or + "--transform Pruning" or + "--transform LowLatency2[use_const_initializer=False]" or + "--transform "MakeStateful[param_res_names= + {'input_name_1':'output_name_1','input_name_2':'output_name_2'}]"" + Available transformations: "LowLatency2", "MakeStateful", "Pruning" + + --transformations_config + Use the configuration file with transformations description or pass + object derived from BaseExtension class. Transformations file can + be specified as relative path from the current directory, as absolute + path or as relative path from the mo root directory. + --silent + Prevent any output messages except those that correspond to log level + equals ERROR, that can be set with the following option: --log_level. + By default, log level is already ERROR. + --log_level + Logger level of logging massages from MO. + Expected one of ['CRITICAL', 'ERROR', 'WARN', 'WARNING', 'INFO', + 'DEBUG', 'NOTSET']. + --version + Version of Model Optimizer + --progress + Enable model conversion progress display. + --stream_output + Switch model conversion progress display to a multiline mode. + --share_weights + Map memory of weights instead reading files or share memory from input + model. + Currently, mapping feature is provided only for ONNX models + that do not require fallback to the legacy ONNX frontend for the conversion. + + + PaddlePaddle-specific parameters: + --example_output + Sample of model output in original framework. For PaddlePaddle it can + be Paddle Variable. + + TensorFlow*-specific parameters: + --input_model_is_text + TensorFlow*: treat the input model file as a text protobuf format. If + not specified, the Model Optimizer treats it as a binary file by default. + + --input_checkpoint + TensorFlow*: variables file to load. + --input_meta_graph + Tensorflow*: a file with a meta-graph of the model before freezing + --saved_model_dir + TensorFlow*: directory with a model in SavedModel format of TensorFlow + 1.x or 2.x version. + --saved_model_tags + Group of tag(s) of the MetaGraphDef to load, in string format, separated + by ','. For tag-set contains multiple tags, all tags must be passed in. + + --tensorflow_custom_operations_config_update + TensorFlow*: update the configuration file with node name patterns + with input/output nodes information. + --tensorflow_object_detection_api_pipeline_config + TensorFlow*: path to the pipeline configuration file used to generate + model created with help of Object Detection API. + --tensorboard_logdir + TensorFlow*: dump the input graph to a given directory that should be + used with TensorBoard. + --tensorflow_custom_layer_libraries + TensorFlow*: comma separated list of shared libraries with TensorFlow* + custom operations implementation. + + MXNet-specific parameters: + --input_symbol + Symbol file (for example, model-symbol.json) that contains a topology + structure and layer attributes + --nd_prefix_name + Prefix name for args.nd and argx.nd files. + --pretrained_model_name + Name of a pretrained MXNet model without extension and epoch number. + This model will be merged with args.nd and argx.nd files + --save_params_from_nd + Enable saving built parameters file from .nd files + --legacy_mxnet_model + Enable MXNet loader to make a model compatible with the latest MXNet + version. Use only if your model was trained with MXNet version lower + than 1.0.0 + --enable_ssd_gluoncv + Enable pattern matchers replacers for converting gluoncv ssd topologies. + + + Caffe*-specific parameters: + --input_proto + Deploy-ready prototxt file that contains a topology structure and + layer attributes + --caffe_parser_path + Path to Python Caffe* parser generated from caffe.proto + --k + Path to CustomLayersMapping.xml to register custom layers + --disable_omitting_optional + Disable omitting optional attributes to be used for custom layers. + Use this option if you want to transfer all attributes of a custom layer + to IR. Default behavior is to transfer the attributes with default values + and the attributes defined by the user to IR. + --enable_flattening_nested_params + Enable flattening optional params to be used for custom layers. Use + this option if you want to transfer attributes of a custom layer to IR + with flattened nested parameters. Default behavior is to transfer + the attributes without flattening nested parameters. + + Kaldi-specific parameters: + --counts + Path to the counts file + --remove_output_softmax + Removes the SoftMax layer that is the output layer + --remove_memory + Removes the Memory layer and use additional inputs outputs instead + + + + +Fetching example models +----------------------- + + + +This notebook uses two models for conversion examples: + +- `Distilbert `__ + NLP model from Hugging Face +- `Resnet50 `__ + CV classification model from torchvision + +.. code:: ipython3 + + from pathlib import Path + + # create a directory for models files + MODEL_DIRECTORY_PATH = Path("model") + MODEL_DIRECTORY_PATH.mkdir(exist_ok=True) + +Fetch +`distilbert `__ +NLP model from Hugging Face and export it in ONNX format: + +.. code:: ipython3 + + from transformers import AutoModelForSequenceClassification, AutoTokenizer + from transformers.onnx import export, FeaturesManager + + + ONNX_NLP_MODEL_PATH = MODEL_DIRECTORY_PATH / "distilbert.onnx" + + # download model + hf_model = AutoModelForSequenceClassification.from_pretrained( + "distilbert-base-uncased-finetuned-sst-2-english" + ) + # initialize tokenizer + tokenizer = AutoTokenizer.from_pretrained( + "distilbert-base-uncased-finetuned-sst-2-english" + ) + + # get model onnx config function for output feature format sequence-classification + model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise( + hf_model, feature="sequence-classification" + ) + # fill onnx config based on pytorch model config + onnx_config = model_onnx_config(hf_model.config) + + # export to onnx format + export( + preprocessor=tokenizer, + model=hf_model, + config=onnx_config, + opset=onnx_config.default_onnx_opset, + output=ONNX_NLP_MODEL_PATH, + ) + + +.. parsed-literal:: + + 2024-01-25 23:12:13.362144: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. + 2024-01-25 23:12:13.397165: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. + To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. + + +.. parsed-literal:: + + 2024-01-25 23:12:14.029251: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT + + +.. parsed-literal:: + + /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. + mask, torch.tensor(torch.finfo(scores.dtype).min) + + + + +.. parsed-literal:: + + (['input_ids', 'attention_mask'], ['logits']) + + + +Fetch +`Resnet50 `__ +CV classification model from torchvision: + +.. code:: ipython3 + + from torchvision.models import resnet50, ResNet50_Weights + + + # create model object + pytorch_model = resnet50(weights=ResNet50_Weights.DEFAULT) + # switch model from training to inference mode + pytorch_model.eval() + + + + +.. parsed-literal:: + + ResNet( + (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) + (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False) + (layer1): Sequential( + (0): Bottleneck( + (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + (downsample): Sequential( + (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): Bottleneck( + (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (2): Bottleneck( + (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + ) + (layer2): Sequential( + (0): Bottleneck( + (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + (downsample): Sequential( + (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False) + (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): Bottleneck( + (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (2): Bottleneck( + (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (3): Bottleneck( + (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + ) + (layer3): Sequential( + (0): Bottleneck( + (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + (downsample): Sequential( + (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False) + (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): Bottleneck( + (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (2): Bottleneck( + (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (3): Bottleneck( + (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (4): Bottleneck( + (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (5): Bottleneck( + (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + ) + (layer4): Sequential( + (0): Bottleneck( + (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + (downsample): Sequential( + (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False) + (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): Bottleneck( + (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + (2): Bottleneck( + (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) + (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) + (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (relu): ReLU(inplace=True) + ) + ) + (avgpool): AdaptiveAvgPool2d(output_size=(1, 1)) + (fc): Linear(in_features=2048, out_features=1000, bias=True) + ) + + + +Convert PyTorch model to ONNX format: + +.. code:: ipython3 + + import torch + import warnings + + + ONNX_CV_MODEL_PATH = MODEL_DIRECTORY_PATH / "resnet.onnx" + + if ONNX_CV_MODEL_PATH.exists(): + print(f"ONNX model {ONNX_CV_MODEL_PATH} already exists.") + else: + with warnings.catch_warnings(): + warnings.filterwarnings("ignore") + torch.onnx.export( + model=pytorch_model, args=torch.randn(1, 3, 780, 520), f=ONNX_CV_MODEL_PATH + ) + print(f"ONNX model exported to {ONNX_CV_MODEL_PATH}") + + +.. parsed-literal:: + + ONNX model model/resnet.onnx already exists. + + +Basic conversion +---------------- + + + +To convert a model to OpenVINO IR, use the following command: + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/distilbert.onnx --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + # mo.convert_model returns an openvino.runtime.Model object + ov_model = mo.convert_model(ONNX_NLP_MODEL_PATH) + + # then model can be serialized to *.xml & *.bin files + from openvino.runtime import serialize + + serialize(ov_model, xml_path=MODEL_DIRECTORY_PATH / "distilbert.xml") + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +Model conversion parameters +--------------------------- + + + +Both Python conversion API and Model Optimizer command-line tool provide +the following capabilities: \* overriding original input shapes for +model conversion with ``input`` and ``input_shape`` parameters. `Setting +Input Shapes +guide `__. +\* cutting off unwanted parts of a model (such as unsupported operations +and training sub-graphs) using the ``input`` and ``output`` parameters +to define new inputs and outputs of the converted model. `Cutting Off +Parts of a Model +guide `__. +\* inserting additional input pre-processing sub-graphs into the +converted model by using the ``mean_values``, ``scales_values``, +``layout``, and other parameters. `Embedding Preprocessing Computation +article `__. +\* compressing the model weights (for example, weights for convolutions +and matrix multiplications) to FP16 data type using ``compress_to_fp16`` +compression parameter. `Compression of a Model to FP16 +guide `__. + +If the out-of-the-box conversion (only the ``input_model`` parameter is +specified) is not successful, it may be required to use the parameters +mentioned above to override input shapes and cut the model. + +Setting Input Shapes +~~~~~~~~~~~~~~~~~~~~ + + + +Model conversion is supported for models with dynamic input shapes that +contain undefined dimensions. However, if the shape of data is not going +to change from one inference request to another, it is recommended to +set up static shapes (when all dimensions are fully defined) for the +inputs. Doing it at this stage, instead of during inference in runtime, +can be beneficial in terms of performance and memory consumption. To set +up static shapes, model conversion API provides the ``input`` and +``input_shape`` parameters. + +For more information refer to `Setting Input Shapes +guide `__. + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,128],[1,128] --output_dir model + + # alternatively + ! mo --input_model model/distilbert.onnx --input input_ids[1,128],attention_mask[1,128] --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model( + ONNX_NLP_MODEL_PATH, + input=["input_ids", "attention_mask"], + input_shape=[[1, 128], [1, 128]], + ) + + # alternatively specify input shapes, using the input parameter + ov_model = mo.convert_model( + ONNX_NLP_MODEL_PATH, input=[("input_ids", [1, 128]), ("attention_mask", [1, 128])] + ) + +The input_shape parameter allows overriding original input shapes to +ones compatible with a given model. Dynamic shapes, i.e. with dynamic +dimensions, can be replaced in the original model with static shapes for +the converted model, and vice versa. The dynamic dimension can be marked +in the model conversion API parameter as ``-1`` or ``?``. For example, +launch model conversion for the ONNX Bert model and specify a dynamic +sequence length dimension for inputs: + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,-1],[1,-1] --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model( + ONNX_NLP_MODEL_PATH, + input=["input_ids", "attention_mask"], + input_shape=[[1, -1], [1, -1]], + ) + +To optimize memory consumption for models with undefined dimensions in +runtime, model conversion API provides the capability to define +boundaries of dimensions. The boundaries of undefined dimensions can be +specified with ellipsis. For example, launch model conversion for the +ONNX Bert model and specify a boundary for the sequence length +dimension: + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/distilbert.onnx --input input_ids,attention_mask --input_shape [1,10..128],[1,10..128] --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model( + ONNX_NLP_MODEL_PATH, + input=["input_ids", "attention_mask"], + input_shape=[[1, "10..128"], [1, "10..128"]], + ) + +Cutting Off Parts of a Model +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +The following examples show when model cutting is useful or even +required: + +- A model has pre- or post-processing parts that cannot be translated + to existing OpenVINO operations. +- A model has a training part that is convenient to be kept in the + model but not used during inference. +- A model is too complex to be converted at once because it contains + many unsupported operations that cannot be easily implemented as + custom layers. +- A problem occurs with model conversion or inference in OpenVINO + Runtime. To identify the issue, limit the conversion scope by an + iterative search for problematic areas in the model. +- A single custom layer or a combination of custom layers is isolated + for debugging purposes. + +For a more detailed description, refer to the `Cutting Off Parts of a +Model +guide `__. + +.. code:: ipython3 + + # Model Optimizer CLI + + # cut at the end + ! mo --input_model model/distilbert.onnx --output /classifier/Gemm --output_dir model + + + # cut from the beginning + ! mo --input_model model/distilbert.onnx --input /distilbert/embeddings/LayerNorm/Add_1,attention_mask --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/distilbert.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + # cut at the end + ov_model = mo.convert_model(ONNX_NLP_MODEL_PATH, output="/classifier/Gemm") + + # cut from the beginning + ov_model = mo.convert_model( + ONNX_NLP_MODEL_PATH, + input=["/distilbert/embeddings/LayerNorm/Add_1", "attention_mask"], + ) + +Embedding Preprocessing Computation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +Input data for inference can be different from the training dataset and +requires additional preprocessing before inference. To accelerate the +whole pipeline, including preprocessing and inference, model conversion +API provides special parameters such as ``mean_values``, +``scale_values``, ``reverse_input_channels``, and ``layout``. Based on +these parameters, model conversion API generates OpenVINO IR with +additionally inserted sub-graphs to perform the defined preprocessing. +This preprocessing block can perform mean-scale normalization of input +data, reverting data along channel dimension, and changing the data +layout. For more information on preprocessing, refer to the `Embedding +Preprocessing Computation +article `__. + +Specifying Layout +^^^^^^^^^^^^^^^^^ + + + +Layout defines the meaning of dimensions in a shape and can be specified +for both inputs and outputs. Some preprocessing requires to set input +layouts, for example, setting a batch, applying mean or scales, and +reversing input channels (BGR<->RGB). For the layout syntax, check the +`Layout API +overview `__. +To specify the layout, you can use the layout option followed by the +layout value. + +The following command specifies the ``NCHW`` layout for a Pytorch +Resnet50 model that was exported to the ONNX format: + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/resnet.onnx --layout nchw --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, layout="nchw") + +Changing Model Layout +^^^^^^^^^^^^^^^^^^^^^ + + + +Changing the model layout may be necessary if it differs from the one +presented by input data. Use either ``layout`` or ``source_layout`` with +``target_layout`` to change the layout. + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/resnet.onnx --layout "nchw->nhwc" --output_dir model + + # alternatively use source_layout and target_layout parameters + ! mo --input_model model/resnet.onnx --source_layout nchw --target_layout nhwc --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, layout="nchw->nhwc") + + # alternatively use source_layout and target_layout parameters + ov_model = mo.convert_model( + ONNX_CV_MODEL_PATH, source_layout="nchw", target_layout="nhwc" + ) + +Specifying Mean and Scale Values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + + +Model conversion API has the following parameters to specify the values: +``mean_values``, ``scale_values``, ``scale``. Using these parameters, +model conversion API embeds the corresponding preprocessing block for +mean-value normalization of the input data and optimizes this block so +that the preprocessing takes negligible time for inference. + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/resnet.onnx --mean_values [123,117,104] --scale 255 --output_dir model + + ! mo --input_model model/resnet.onnx --mean_values [123,117,104] --scale_values [255,255,255] --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, mean_values=[123, 117, 104], scale=255) + + ov_model = mo.convert_model( + ONNX_CV_MODEL_PATH, mean_values=[123, 117, 104], scale_values=[255, 255, 255] + ) + +Reversing Input Channels +^^^^^^^^^^^^^^^^^^^^^^^^ + + + +Sometimes, input images for your application can be of the ``RGB`` (or +``BGR``) format, and the model is trained on images of the ``BGR`` (or +``RGB``) format, which is in the opposite order of color channels. In +this case, it is important to preprocess the input images by reverting +the color channels before inference. + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/resnet.onnx --reverse_input_channels --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, reverse_input_channels=True) + +Compressing a Model to FP16 +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + +Optionally all relevant floating-point weights can be compressed to FP16 +data type during the model conversion, creating a compressed FP16 model. +This smaller model occupies about half of the original space in the file +system. While the compression may introduce a drop in accuracy, for most +models, this decrease is negligible. + +.. code:: ipython3 + + # Model Optimizer CLI + + ! mo --input_model model/resnet.onnx --compress_to_fp16=True --output_dir model + + +.. parsed-literal:: + + huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... + To disable this warning, you can either: + - Avoid using `tokenizers` before the fork if possible + - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) + + +.. parsed-literal:: + + [ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False. + Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html + [ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11. + Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html + [ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API. + Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html + + +.. parsed-literal:: + + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.xml + [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-598/.workspace/scm/ov-notebook/notebooks/121-convert-to-openvino/model/resnet.bin + + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(ONNX_CV_MODEL_PATH, compress_to_fp16=True) + +Convert Models Represented as Python Objects +-------------------------------------------- + + + +Python conversion API can pass Python model objects, such as a Pytorch +model or TensorFlow Keras model directly, without saving them into files +and without leaving the training environment (Jupyter Notebook or +training scripts). + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model(pytorch_model) + + +.. parsed-literal:: + + WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11. + + +``convert_model()`` accepts all parameters available in the MO +command-line tool. Parameters can be specified by Python classes or +string analogs, similar to the command-line tool. + +.. code:: ipython3 + + # Python conversion API + from openvino.tools import mo + + + ov_model = mo.convert_model( + pytorch_model, + input_shape=[1, 3, 100, 100], + mean_values=[127, 127, 127], + layout="nchw", + ) + + ov_model = mo.convert_model(pytorch_model, source_layout="nchw", target_layout="nhwc") + + ov_model = mo.convert_model( + pytorch_model, compress_to_fp16=True, reverse_input_channels=True + ) diff --git a/docs/notebooks/122-speech-recognition-quantization-wav2vec2-with-output.rst b/docs/notebooks/122-speech-recognition-quantization-wav2vec2-with-output.rst index c6cacb354a2..c08741175df 100644 --- a/docs/notebooks/122-speech-recognition-quantization-wav2vec2-with-output.rst +++ b/docs/notebooks/122-speech-recognition-quantization-wav2vec2-with-output.rst @@ -48,17 +48,19 @@ and has the following differences: The steps for the quantization with accuracy control are described below. -**Table of contents:** +Table of contents: +^^^^^^^^^^^^^^^^^^ - -- `Imports <#>`__ -- `Prepare the Model <#>`__ -- `Prepare LibriSpeech Dataset <#>`__ -- `Prepare calibration and validation datasets <#>`__ -- `Prepare validation function <#>`__ -- `Run quantization with accuracy control <#>`__ -- `Model Usage Example <#>`__ -- `Compare Performance of the Original and Quantized Models <#>`__ +- `Imports <#imports>`__ +- `Prepare the Model <#prepare-the-model>`__ +- `Prepare LibriSpeech Dataset <#prepare-librispeech-dataset>`__ +- `Prepare calibration dataset <#prepare-calibration-dataset>`__ +- `Prepare validation function <#prepare-validation-function>`__ +- `Run quantization with accuracy + control <#run-quantization-with-accuracy-control>`__ +- `Model Usage Example <#model-usage-example>`__ +- `Compare Accuracy of the Original and Quantized + Models <#compare-accuracy-of-the-original-and-quantized-models>`__ .. code:: ipython3 @@ -67,13 +69,15 @@ below. %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu soundfile librosa transformers torch datasets torchmetrics Imports ----------------------------------- +------- + + .. code:: ipython3 import numpy as np import torch - + from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor @@ -86,14 +90,16 @@ Imports Prepare the Model ----------------------------------- +----------------- -For instantiating PyTorch model class, -we should use ``Wav2Vec2ForCTC.from_pretrained`` method with providing -model ID for downloading from HuggingFace hub. Model weights and -configuration files will be downloaded automatically in first time -usage. Keep in mind that downloading the files can take several minutes -and depends on your internet connection. + + +For instantiating PyTorch model class, we should use +``Wav2Vec2ForCTC.from_pretrained`` method with providing model ID for +downloading from HuggingFace hub. Model weights and configuration files +will be downloaded automatically in first time usage. Keep in mind that +downloading the files can take several minutes and depends on your +internet connection. Additionally, we can create processor class which is responsible for model specific pre- and post-processing steps. @@ -102,8 +108,8 @@ model specific pre- and post-processing steps. BATCH_SIZE = 1 MAX_SEQ_LENGTH = 30480 - - + + torch_model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h", ctc_loss_reduction="mean") processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h") @@ -119,8 +125,8 @@ Convert it to the OpenVINO Intermediate Representation (OpenVINO IR) .. code:: ipython3 import openvino as ov - - + + default_input = torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float) ov_model = ov.convert_model(torch_model, example_input=default_input) @@ -140,7 +146,9 @@ Convert it to the OpenVINO Intermediate Representation (OpenVINO IR) Prepare LibriSpeech Dataset ---------------------------------------------------- +--------------------------- + + For demonstration purposes, we will use short dummy version of LibriSpeech dataset - ``patrickvonplaten/librispeech_asr_dummy`` to @@ -151,20 +159,20 @@ dataset. .. code:: ipython3 from datasets import load_dataset - - + + dataset = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") test_sample = dataset[0]["audio"] - - + + # define preprocessing function for converting audio to input values for model def map_to_input(batch): preprocessed_signal = processor(batch["audio"]["array"], return_tensors="pt", padding="longest", sampling_rate=batch['audio']['sampling_rate']) input_values = preprocessed_signal.input_values batch['input_values'] = input_values return batch - - + + # apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore dataset = dataset.map(map_to_input, batched=False, remove_columns=["audio"]) @@ -176,13 +184,15 @@ dataset. Prepare calibration dataset ----------------------------------- +--------------------------- + + .. code:: ipython3 import nncf - - + + def transform_fn(data_item): """ Extract the model's input from the data item. @@ -190,8 +200,8 @@ Prepare calibration dataset This function should be passed when the data item cannot be used as model's input. """ return np.array(data_item["input_values"]) - - + + calibration_dataset = nncf.Dataset(dataset, transform_fn) @@ -201,7 +211,9 @@ Prepare calibration dataset Prepare validation function ----------------------------------- +--------------------------- + + Define the validation function. @@ -209,8 +221,8 @@ Define the validation function. from torchmetrics import WordErrorRate from tqdm.notebook import tqdm - - + + def validation_fn(model, dataset): """ Calculate and returns a metric for the model. @@ -222,35 +234,31 @@ Define the validation function. logits = model(np.array(sample['input_values']))[output] predicted_ids = np.argmax(logits, axis=-1) transcription = processor.batch_decode(torch.from_numpy(predicted_ids)) - + # update metric on sample result wer.update(transcription, [sample['text']]) - + result = wer.compute() - + return 1 - result Run quantization with accuracy control ---------------------------------------------------- +-------------------------------------- -You should provide -the calibration dataset and the validation dataset. It can be the same -dataset. -- parameter ``max_drop`` defines the accuracy drop threshold. - The quantization process stops when the degradation of accuracy metric - on the validation dataset is less than the ``max_drop``. The default - value is 0.01. NNCF will stop the quantization and report an error if - the ``max_drop`` value can’t be reached. -- ``drop_type`` defines how the - accuracy drop will be calculated: ABSOLUTE (used by default) or - RELATIVE. - -- ``ranking_subset_size`` - size of a subset that is used to - rank layers by their contribution to the accuracy drop. Default value is - 300, and the more samples it has the better ranking, potentially. Here - we use the value 25 to speed up the execution. +You should provide the calibration dataset and the validation dataset. +It can be the same dataset. - parameter ``max_drop`` defines the +accuracy drop threshold. The quantization process stops when the +degradation of accuracy metric on the validation dataset is less than +the ``max_drop``. The default value is 0.01. NNCF will stop the +quantization and report an error if the ``max_drop`` value can’t be +reached. - ``drop_type`` defines how the accuracy drop will be +calculated: ABSOLUTE (used by default) or RELATIVE. - +``ranking_subset_size`` - size of a subset that is used to rank layers +by their contribution to the accuracy drop. Default value is 300, and +the more samples it has the better ranking, potentially. Here we use the +value 25 to speed up the execution. **NOTE**: Execution can take tens of minutes and requires up to 10 GB of free memory @@ -259,7 +267,7 @@ dataset. from nncf.quantization.advanced_parameters import AdvancedAccuracyRestorerParameters from nncf.parameters import ModelType - + quantized_model = nncf.quantize_with_accuracy_control( ov_model, calibration_dataset=calibration_dataset, @@ -316,13 +324,13 @@ dataset. INFO:nncf:Calculating ranking score for groups of quantizers INFO:nncf:Elapsed Time: 00:04:36 INFO:nncf:Changing the scope of quantizer nodes was started - INFO:nncf:Reverted 1 operations to the floating-point precision: + INFO:nncf:Reverted 1 operations to the floating-point precision: __module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution_11 INFO:nncf:Accuracy drop with the new quantization scope is 0.06173914670944214 (DropType.ABSOLUTE) - INFO:nncf:Reverted 1 operations to the floating-point precision: + INFO:nncf:Reverted 1 operations to the floating-point precision: __module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution_10 INFO:nncf:Accuracy drop with the new quantization scope is 0.010434746742248535 (DropType.ABSOLUTE) - INFO:nncf:Reverted 1 operations to the floating-point precision: + INFO:nncf:Reverted 1 operations to the floating-point precision: __module.wav2vec2.feature_extractor.conv_layers.3.conv/aten::_convolution/Convolution_12 INFO:nncf:Algorithm completed: achieved required accuracy drop 0.006956517696380615 (DropType.ABSOLUTE) INFO:nncf:3 out of 94 were reverted back to the floating-point precision: @@ -332,13 +340,15 @@ dataset. Model Usage Example ----------------------------------- +------------------- + + .. code:: ipython3 import IPython.display as ipd - - + + ipd.Audio(test_sample["array"], rate=16000) @@ -346,7 +356,7 @@ Model Usage Example .. raw:: html - +