natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] device section restructure (#22321)

This commit is contained in:
Karol Blaszczak 2024-02-13 13:17:29 +01:00 committed by GitHub
parent 19ba148b4f
commit 9c406346d1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
31 changed files with 398 additions and 385 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
docs/articles_en/about_openvino/additional_resources/glossary.rst
View File

@ -114,16 +114,18 @@ Glossary of terms used in OpenVINO™
| Represents data element type. For example, f32 is 32-bit floating point, f16 is 16-bit floating point.
| *plugin / Inference Device / Inference Mode*
| OpenVINO makes hardware available for inference based on several core components. They used to be called "plugins" in earlier versions of documentation and you may still find this term in some articles. Because of their role in the software, they are now referred to as Devices and Modes ("virtual" devices). For a detailed description of the concept, refer to [Inference Modes](@ref openvino_docs_Runtime_Inference_Modes_Overview) and [Inference Devices](@ref openvino_docs_OV_UG_Working_with_devices).
| OpenVINO makes hardware available for inference based on several core components.
They used to be called "plugins" in earlier versions of documentation and you may
still find this term in some articles. Because of their role in the software,
they are now referred to as Devices and Modes ("virtual" devices). For a detailed
description of the concept, refer to
:doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
| *Tensor*
| A memory container used for storing inputs and outputs of the model, as well as weights and biases of the operations.
| A memory container used for storing inputs and outputs of the model, as well as
weights and biases of the operations.
See Also
#################################################
* :doc:`Available Operations Sets <openvino_docs_ops_opset>`
* :doc:`Terminology <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`

6
docs/articles_en/about_openvino/compatibility_and_support.rst
View File

@ -9,16 +9,16 @@ Compatibility and Support
:hidden:
openvino_supported_models
openvino_docs_OV_UG_supported_plugins_Supported_Devices
openvino_supported_devices
openvino_resources_supported_operations
openvino_resources_supported_operations_frontend
:doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` - compatibility information for supported hardware accelerators.
:doc:`Supported Devices <openvino_supported_devices>` - compatibility information for supported hardware accelerators.
:doc:`Supported Models <openvino_supported_models>` - a table of models officially supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations>` - a listing of framework layers supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations>` - a listing of framework layers supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations_frontend>` - a listing of layers supported by OpenVINO inference devices.

109
docs/articles_en/about_openvino/compatibility_and_support/Supported_Devices.rst
View File

@ -1,55 +1,89 @@
.. {#openvino_docs_OV_UG_supported_plugins_Supported_Devices}
Supported Devices
=================
.. {#openvino_supported_devices}
Inference Device Support
========================
.. meta::
:description: Check the list of officially supported models in Intel®
Distribution of OpenVINO™ toolkit.
:description: Check the list of devices used by OpenVINO to run inference
of deep learning models.
OpenVINO enables you to implement its inference capabilities in your own software,
utilizing various hardware. It currently supports the following processing units
(for more details, see :doc:`system requirements <system_requirements>`):
* :doc:`CPU <openvino_docs_OV_UG_supported_plugins_CPU>`
* :doc:`GPU <openvino_docs_OV_UG_supported_plugins_GPU>`
.. note::
With OpenVINO 2024 release, GNA has been deprecated together
with the hardware being discontinued in future CPU solutions.
With OpenVINO™ 2023.0 release, support has been cancelled for:
- Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X
- Intel® Vision Accelerator Design with Intel® Movidius™
To keep using the MYRIAD and HDDL plugins with your hardware, revert to the OpenVINO 2022.3 LTS release.
The OpenVINO™ runtime enables you to use a selection of devices to run your
deep learning models:
:doc:`CPU <openvino_docs_OV_UG_supported_plugins_CPU>`,
:doc:`GPU <openvino_docs_OV_UG_supported_plugins_GPU>`,
:doc:`NPU <openvino_docs_OV_UG_supported_plugins_NPU>`.
| For their usage guides, see :doc:`Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
| For a detailed list of devices, see :doc:`System Requirements <system_requirements>`.
Beside running inference with a specific device,
OpenVINO offers automated inference management with the following inference modes:
OpenVINO offers the option of running automated inference with the following inference modes:
* :doc:`Automatic Device Selection <openvino_docs_OV_UG_supported_plugins_AUTO>` - automatically selects the best device
available for the given task. It offers many additional options and optimizations, including inference on
multiple devices at the same time.
* :doc:`Heterogeneous Inference <openvino_docs_OV_UG_Hetero_execution>` - enables splitting inference among several devices
automatically, for example, if one device doesn't support certain operations.
* :doc:`Multi-device Inference <openvino_docs_OV_UG_Running_on_multiple_devices>` - executes inference on multiple devices.
Currently, this mode is considered a legacy solution. Using Automatic Device Selection is advised.
* :doc:`Heterogeneous Inference <openvino_docs_OV_UG_Hetero_execution>` - enables splitting inference among several devices
automatically, for example, if one device doesn’t support certain operations.
* :doc:`Automatic Batching <openvino_docs_OV_UG_Automatic_Batching>` - automatically groups inference requests to improve
device utilization.
Devices similar to the ones used for benchmarking can be accessed using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__,
a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution
of OpenVINO™ Toolkit. `Learn more <https://devcloud.intel.com/edge/get_started/devcloud/>`__ or `Register here <https://inteliot.force.com/DevcloudForEdge/s/>`__.
Feature Support and API Coverage
#################################
================================================================================== ======= ========== ===========
Supported Feature CPU GPU NPU
================================================================================== ======= ========== ===========
:doc:`Heterogeneous execution <openvino_docs_OV_UG_Hetero_execution>` Yes Yes No
:doc:`Multi-device execution <openvino_docs_OV_UG_Running_on_multiple_devices>` Yes Yes Partial
:doc:`Automatic batching <openvino_docs_OV_UG_Automatic_Batching>` No Yes No
:doc:`Multi-stream execution <openvino_docs_deployment_optimization_guide_tput>` Yes Yes No
:doc:`Models caching <openvino_docs_OV_UG_Model_caching_overview>` Yes Partial Yes
:doc:`Dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` Yes Partial No
:doc:`Import/Export <openvino_ecosystem>` Yes No Yes
:doc:`Preprocessing acceleration <openvino_docs_OV_UG_Preprocessing_Overview>` Yes Yes No
:doc:`Stateful models <openvino_docs_OV_UG_stateful_models_intro>` Yes No Yes
:doc:`Extensibility <openvino_docs_Extensibility_UG_Intro>` Yes Yes No
================================================================================== ======= ========== ===========
To learn more about each of the supported devices and modes, refer to the sections of:
* :doc:`Inference Device Support <openvino_docs_OV_UG_Working_with_devices>`
* :doc:`Inference Modes <openvino_docs_Runtime_Inference_Modes_Overview>`
+-------------------------+-----------+------------------+-------------------+
| **API Coverage:** | plugin | infer_request | compiled_model |
+=========================+===========+==================+===================+
| CPU | 80.0 % | 100.0 % | 89.74 % |
+-------------------------+-----------+------------------+-------------------+
| CPU_ARM | 80.0 % | 100.0 % | 89.74 % |
+-------------------------+-----------+------------------+-------------------+
| GPU | 84.0 % | 100.0 % | 100.0 % |
+-------------------------+-----------+------------------+-------------------+
| dGPU | 82.0 % | 100.0 % | 100.0 % |
+-------------------------+-----------+------------------+-------------------+
| NPU | 16.0 % | 0.0 % | 10.26 % |
+-------------------------+-----------+------------------+-------------------+
| AUTO | 40.0 % | 100.0 % | 97.44 % |
+-------------------------+-----------+------------------+-------------------+
| BATCH | 26.0 % | 100.0 % | 58.97 % |
+-------------------------+-----------+------------------+-------------------+
| MULTI | 30.0 % | 100.0 % | 58.97 % |
+-------------------------+-----------+------------------+-------------------+
| HETERO | 30.0 % | 99.23 % | 58.97 % |
+-------------------------+-----------+------------------+-------------------+
| || Percentage of API supported by the device, |
| || as of OpenVINO 2023.3, 08 Jan, 2024. |
+-------------------------+-----------+------------------+-------------------+
Devices similar to the ones used for benchmarking can be accessed using
`Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__,
a remote development environment with access to Intel® hardware and the latest versions
of the Intel® Distribution of OpenVINO™ Toolkit.
`Learn more <https://devcloud.intel.com/edge/get_started/devcloud/>`__ or
`Register here <https://inteliot.force.com/DevcloudForEdge/s/>`__.
For setting up a relevant configuration, refer to the
:doc:`Integrate with Customer Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
@ -57,5 +91,14 @@ topic (step 3 "Configure input and output").
.. note::
With OpenVINO 2024.0 release, support for GNA has been discontinued. To keep using it
in your solutions, revert to the 2023.3 (LTS) version.
With OpenVINO™ 2023.0 release, support has been cancelled for:
- Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X
- Intel® Vision Accelerator Design with Intel® Movidius™
To keep using the MYRIAD and HDDL plugins with your hardware,
revert to the OpenVINO 2022.3 (LTS) version.

86
docs/articles_en/about_openvino/performance_benchmarks/Getting_performance_numbers.rst
View File

@ -5,8 +5,8 @@ Getting Performance Numbers
This guide explains how to use the benchmark_app to get performance numbers. It also explains how the performance
numbers are reflected through internal inference performance counters and execution graphs. It also includes
This guide explains how to use the benchmark_app to get performance numbers. It also explains how the performance
numbers are reflected through internal inference performance counters and execution graphs. It also includes
information on using ITT and Intel® VTune™ Profiler to get performance insights.
Test performance with the benchmark_app
@ -19,7 +19,7 @@ The Python one is part of OpenVINO Runtime installation, while C++ is available
For a detailed description, see: :doc:`benchmark_app <openvino_sample_benchmark_tool>`.
Make sure to install the latest release package with support for frameworks of the models you want to test.
For the most reliable performance benchmarks, :doc:`prepare the model for use with OpenVINO <openvino_docs_model_processing_introduction>`.
For the most reliable performance benchmarks, :doc:`prepare the model for use with OpenVINO <openvino_docs_model_processing_introduction>`.
Running the benchmark application
@ -32,10 +32,10 @@ The benchmark_app includes a lot of device-specific options, but the primary usa
benchmark_app -m <model> -d <device> -i <input>
Each of the :doc:`OpenVINO supported devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` offers
Each of the :doc:`OpenVINO supported devices <openvino_supported_devices>` offers
performance settings that contain command-line equivalents in the Benchmark app.
While these settings provide really low-level control for the optimal model performance on the *specific* device,
While these settings provide really low-level control for the optimal model performance on the *specific* device,
it is recommended to always start performance evaluation with the :doc:`OpenVINO High-Level Performance Hints <openvino_docs_OV_UG_Performance_Hints>` first, like so:
.. code-block:: sh
@ -55,32 +55,32 @@ Additional benchmarking considerations
When evaluating performance of a model with OpenVINO Runtime, it is required to measure a proper set of operations.
- Avoid including one-time costs such as model loading.
- Track operations that occur outside OpenVINO Runtime (such as video decoding) separately.
- Track operations that occur outside OpenVINO Runtime (such as video decoding) separately.
.. note::
Some image pre-processing can be baked into OpenVINO IR and accelerated accordingly. For more information,
refer to :doc:`Embedding Pre-processing <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>` and
Some image pre-processing can be baked into OpenVINO IR and accelerated accordingly. For more information,
refer to :doc:`Embedding Pre-processing <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>` and
:doc:`General Runtime Optimizations <openvino_docs_deployment_optimization_guide_common>`.
2 - Try to Get Credible Data
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Performance conclusions should be build upon reproducible data. As for the performance measurements, they should
be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly
Performance conclusions should be build upon reproducible data. As for the performance measurements, they should
be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly
slower than the subsequent ones, an aggregated value can be used for the execution time for final projections:
- If the warm-up run does not help or execution time still varies, you can try running a large number of iterations
- If the warm-up run does not help or execution time still varies, you can try running a large number of iterations
and then average or find a mean of the results.
- If the time values range too much, consider geomean.
- Be aware of the throttling and other power oddities. A device can exist in one of several different power states.
When optimizing your model, consider fixing the device frequency for better performance data reproducibility.
- Be aware of the throttling and other power oddities. A device can exist in one of several different power states.
When optimizing your model, consider fixing the device frequency for better performance data reproducibility.
However, the end-to-end (application) benchmarking should also be performed under real operational conditions.
3 - Compare Performance with Native/Framework Code
3 - Compare Performance with Native/Framework Code
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When comparing the OpenVINO Runtime performance with the framework or another reference code, make sure that both versions are as similar as possible:
@ -96,61 +96,61 @@ Internal Inference Performance Counters and Execution Graphs
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
More detailed insights into inference performance breakdown can be achieved with device-specific performance counters and/or execution graphs.
Both :doc:`C++ and Python <openvino_sample_benchmark_tool>`
Both :doc:`C++ and Python <openvino_sample_benchmark_tool>`
versions of the *benchmark_app* support a ``-pc`` command-line parameter that outputs internal execution breakdown.
For example, the table shown below is part of performance counters for quantized
`TensorFlow implementation of ResNet-50 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf>`__
For example, the table shown below is part of performance counters for quantized
`TensorFlow implementation of ResNet-50 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf>`__
model inference on :doc:`CPU Plugin <openvino_docs_OV_UG_supported_plugins_CPU>`.
Keep in mind that since the device is CPU, the ``realTime`` wall clock and the ``cpu`` time layers are the same.
Information about layer precision is also stored in the performance counters.
Keep in mind that since the device is CPU, the ``realTime`` wall clock and the ``cpu`` time layers are the same.
Information about layer precision is also stored in the performance counters.
=========================================================== ============= ============== ===================== ================= ==============
layerName execStatus layerType execType realTime (ms) cpuTime (ms)
layerName execStatus layerType execType realTime (ms) cpuTime (ms)
=========================================================== ============= ============== ===================== ================= ==============
resnet\_model/batch\_normalization\_15/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.377 0.377
resnet\_model/conv2d\_16/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_16/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_I8 0.499 0.499
resnet\_model/conv2d\_17/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_17/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.399 0.399
resnet\_model/add\_4/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/add\_4 NOT\_RUN Eltwise undef 0 0
resnet\_model/add\_5/fq\_input\_1 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_15/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.377 0.377
resnet\_model/conv2d\_16/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_16/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_I8 0.499 0.499
resnet\_model/conv2d\_17/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_17/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.399 0.399
resnet\_model/add\_4/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/add\_4 NOT\_RUN Eltwise undef 0 0
resnet\_model/add\_5/fq\_input\_1 NOT\_RUN FakeQuantize undef 0 0
=========================================================== ============= ============== ===================== ================= ==============
| The ``execStatus`` column of the table includes the following possible values:
| - ``EXECUTED`` - the layer was executed by standalone primitive.
| - ``NOT_RUN`` - the layer was not executed by standalone primitive or was fused with another operation and executed in another layer primitive.
|
| - ``NOT_RUN`` - the layer was not executed by standalone primitive or was fused with another operation and executed in another layer primitive.
|
| The ``execType`` column of the table includes inference primitives with specific suffixes. The layers could have the following marks:
| - The ``I8`` suffix is for layers that had 8-bit data type input and were computed in 8-bit precision.
| - The ``FP32`` suffix is for layers computed in 32-bit precision.
|
| All ``Convolution`` layers are executed in ``int8`` precision. The rest of the layers are fused into Convolutions using post-operation optimization,
as described in :doc:`CPU Device <openvino_docs_OV_UG_supported_plugins_CPU>`. This contains layer names
|
| All ``Convolution`` layers are executed in ``int8`` precision. The rest of the layers are fused into Convolutions using post-operation optimization,
as described in :doc:`CPU Device <openvino_docs_OV_UG_supported_plugins_CPU>`. This contains layer names
(as seen in OpenVINO IR), type of the layer, and execution statistics.
Both *benchmark_app* versions also support the ``exec_graph_path`` command-line option. It requires OpenVINO to output the same execution
Both *benchmark_app* versions also support the ``exec_graph_path`` command-line option. It requires OpenVINO to output the same execution
statistics per layer, but in the form of plugin-specific `Netron-viewable <https://netron.app/>`__ graph to the specified file.
Especially when performance-debugging the :doc:`latency <openvino_docs_deployment_optimization_guide_latency>`, note that the counters
do not reflect the time spent in the ``plugin/device/driver/etc`` queues. If the sum of the counters is too different from the latency
of an inference request, consider testing with less inference requests. For example, running single
:doc:`OpenVINO stream <openvino_docs_deployment_optimization_guide_tput>` with multiple requests would produce nearly identical
Especially when performance-debugging the :doc:`latency <openvino_docs_deployment_optimization_guide_latency>`, note that the counters
do not reflect the time spent in the ``plugin/device/driver/etc`` queues. If the sum of the counters is too different from the latency
of an inference request, consider testing with less inference requests. For example, running single
:doc:`OpenVINO stream <openvino_docs_deployment_optimization_guide_tput>` with multiple requests would produce nearly identical
counters as running a single inference request, while the actual latency can be quite different.
Lastly, the performance statistics with both performance counters and execution graphs are averaged,
so such data for the :doc:`inputs of dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` should be measured carefully,
Lastly, the performance statistics with both performance counters and execution graphs are averaged,
so such data for the :doc:`inputs of dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` should be measured carefully,
preferably by isolating the specific shape and executing multiple times in a loop, to gather reliable data.
Use ITT to Get Performance Insights
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT).
Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like
`Intel® VTune™ Profiler <https://software.intel.com/en-us/vtune>`__ to get detailed inference performance breakdown and additional
In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT).
Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like
`Intel® VTune™ Profiler <https://software.intel.com/en-us/vtune>`__ to get detailed inference performance breakdown and additional
insights in the application-level performance on the timeline view.

21
docs/articles_en/about_openvino/performance_benchmarks/performance_benchmarks_faq.rst
View File

@ -92,36 +92,35 @@ Performance Information F.A.Q.
* - `ssd-mobilenet-v1-coco <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd_mobilenet_v1_coco>`__
- ssd-mobilenet-V1-coco onnx model
- object detection
- 300x300
- 300x300
* - `ssd-resnet34-1200-onnx <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd-resnet34-1200-onnx>`__
- ssd-resnet34 onnx model
- object detection
- 1200x1200
- 1200x1200
* - `unet-camvid-onnx-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/unet-camvid-onnx-0001>`__
- U-Net
- semantic segmentation
- 368x480
- 368x480
* - `yolo-v3-tiny <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tiny-tf>`__
- YOLO v3 Tiny
- object detection
- 416x416
- 416x416
* - `yolov8n <https://https://github.com/ultralytics/ultralytics>`__
- Yolov8nano
- Yolov8nano
- object detection
- 608x608
.. dropdown:: Where can I purchase the specific hardware used in the benchmarking?
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
For more details, see the :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
documentation. Before purchasing any hardware, you can test and run
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
Before purchasing any hardware, you can test and run models remotely, using
`Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
.. dropdown:: How can I optimize my models for better performance or accuracy?
Set of guidelines and recommendations to optimize models are available in the
Set of guidelines and recommendations to optimize models are available in the
:doc:`optimization guide <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
Join the conversation in the `Community Forum <https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit>`__ for further support.

6
docs/articles_en/documentation/openvino_legacy_features.rst
View File

@ -88,8 +88,7 @@ offering.
| *New solution:* the tool is no longer needed
| *Old solution:* deprecated in OpenVINO 2023.0
|
| Compile tool is now deprecated. If you need to compile a model for inference on
a specific device, use the following script:
| If you need to compile a model for inference on a specific device, use the following script:
.. tab-set::
@ -107,9 +106,6 @@ offering.
:language: cpp
:fragment: [export_compiled_model]
| :doc:`see which devices support import / export <openvino_docs_OV_UG_Working_with_devices>`
| :doc:`Learn more on preprocessing steps <openvino_docs_OV_UG_Preprocessing_Overview>`
| :doc:`See how to integrate and save preprocessing steps into OpenVINO IR <openvino_docs_OV_UG_Preprocess_Usecase_save>`
| **DL Workbench**
| *New solution:* DevCloud version

6
docs/articles_en/documentation/openvino_legacy_features/mo_ovc_transition/legacy_conversion_api/fp16_compression.rst
View File

@ -7,8 +7,8 @@
The code described here has been **deprecated!** Do not use it to avoid working with a legacy solution. It will be kept for some time to ensure backwards compatibility, but **you should not use** it in contemporary applications.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Conversion Parameters <openvino_docs_OV_Converter_UG_Conversion_Options>` article.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Conversion Parameters <openvino_docs_OV_Converter_UG_Conversion_Options>` article.
By default, when IR is saved all relevant floating-point weights are compressed to ``FP16`` data type during model conversion.
It results in creating a "compressed ``FP16`` model", which occupies about half of
the original space in the file system. The compression may introduce a minor drop in accuracy,
@ -37,7 +37,7 @@ To disable compression, use the ``compress_to_fp16=False`` option:
For details on how plugins handle compressed ``FP16`` models, see
:doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`.
:doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
.. note::

6
docs/articles_en/documentation/openvino_legacy_features/mo_ovc_transition/legacy_conversion_api/setting_input_shapes.rst
View File

@ -7,12 +7,12 @@
The code described here has been **deprecated!** Do not use it to avoid working with a legacy solution. It will be kept for some time to ensure backwards compatibility, but **you should not use** it in contemporary applications.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Setting Input Shapes <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Converting_Model>` article.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Setting Input Shapes <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Converting_Model>` article.
With model conversion API you can increase your model's efficiency by providing an additional shape definition, with these two parameters: `input_shape` and `static_shape`.
.. meta::
.. meta::
:description: Learn how to increase the efficiency of a model with MO by providing an additional shape definition with the input_shape and static_shape parameters.
@ -27,7 +27,7 @@ To set up static shapes, model conversion API provides the ``input_shape`` param
For more information on input shapes under runtime, refer to the :doc:`Changing input shapes <openvino_docs_OV_UG_ShapeInference>` guide.
To learn more about dynamic shapes in runtime, refer to the :doc:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>` guide.
The OpenVINO Runtime API may present certain limitations in inferring models with undefined dimensions on some hardware. See the :doc:`Features support matrix <openvino_docs_OV_UG_Working_with_devices>` for reference.
The OpenVINO Runtime API may present certain limitations in inferring models with undefined dimensions on some hardware.
In this case, the ``input_shape`` parameter and the :doc:`reshape method <openvino_docs_OV_UG_ShapeInference>` can help to resolve undefined dimensions.
For example, run model conversion for the TensorFlow MobileNet model with the single input

8
docs/articles_en/openvino_workflow/deployment_intro/local-distribution.rst
View File

@ -111,13 +111,17 @@ As shown in the picture above, some plugin libraries may have OS-specific depend
Libraries for Execution Modes
+++++++++++++++++++++++++++++
The ``HETERO``, ``MULTI``, ``BATCH`` and ``AUTO`` execution modes can also be used by the application explicitly or implicitly. Use the following recommendation scheme to decide whether to add the appropriate libraries to the distribution package:
The ``HETERO``, ``MULTI``, ``BATCH``, and ``AUTO`` execution modes can also be used by the application explicitly or implicitly.
Use the following recommendation scheme to decide whether to add the appropriate libraries to the distribution package:
- If :doc:`AUTO <openvino_docs_OV_UG_supported_plugins_AUTO>` is used explicitly in the application or `ov::Core::compile_model <classov_1_1Core.html#doxid-classov-1-1-core-1a46555f0803e8c29524626be08e7f5c5a>`__ is used without specifying a device, put ``openvino_auto_plugin`` to the distribution.
.. note::
Automatic Device Selection relies on :doc:`inference device plugins <openvino_docs_OV_UG_Working_with_devices>`. If you are not sure which inference devices are available on the target system, put all inference plugin libraries in the distribution. If `ov::device::priorities <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1gae88af90a18871677f39739cb0ef0101e>`__ is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only.
Automatic Device Selection relies on :doc:`inference device plugins <openvino_docs_Runtime_Inference_Modes_Overview>`.
If you are not sure which inference devices are available on the target system, put all inference plugin libraries in the distribution.
If `ov::device::priorities <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1gae88af90a18871677f39739cb0ef0101e>`__
is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only.
- If :doc:`MULTI <openvino_docs_OV_UG_Running_on_multiple_devices>` is used explicitly, put ``openvino_auto_plugin`` in the distribution.
- If :doc:`HETERO <openvino_docs_OV_UG_Hetero_execution>` is either used explicitly or `ov::hint::performance_mode <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1ga2691fe27acc8aa1d1700ad40b6da3ba2>`__ is used with GPU, put ``openvino_hetero_plugin`` in the distribution.

97
docs/articles_en/openvino_workflow/model_preparation.rst
View File

@ -26,43 +26,52 @@ OpenVINO supports the following model formats:
* PaddlePaddle,
* OpenVINO IR.
The easiest way to obtain a model is to download it from an online database, such as `TensorFlow Hub <https://tfhub.dev/>`__, `Hugging Face <https://huggingface.co/>`__,
and `Torchvision models <https://pytorch.org/hub/>`__. Now you have two options:
The easiest way to obtain a model is to download it from an online database, such as
`TensorFlow Hub <https://tfhub.dev/>`__, `Hugging Face <https://huggingface.co/>`__, and
`Torchvision models <https://pytorch.org/hub/>`__. Now you have two options:
* Skip model conversion and :doc:`run inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>` directly from the **TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle** source format. Conversion
will still be performed but it will happen automatically and "under the hood".
This option, while convenient, offers lower performance and stability, as well as
fewer optimization options.
* Skip model conversion and :doc:`run inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
directly from the **TensorFlow, TensorFlow Lite, ONNX, or PaddlePaddle** source format.
Conversion will still be performed but it will happen automatically and "under the hood".
This option, while convenient, offers lower performance and stability, as well as fewer
optimization options.
* Explicitly :doc:`convert the model to OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`.
This approach offers the best possible results and is the recommended one,
especially for production-ready solutions. Consider storing your model in this format to minimize first-inference latency,
perform model optimizations, and save space on your drive, in some cases. Explicit conversion can be done in two ways:
This approach offers the best possible results and is the recommended one, especially for
production-ready solutions. Consider storing your model in this format to minimize
first-inference latency, perform model optimizations, and save space on your drive, in some
cases. Explicit conversion can be done in two ways:
* the `Python API functions <#convert-a-model-with-python-convert-model>`__ (``openvino.convert_model`` and ``openvino.save_model``)
* the `Python API functions <#convert-a-model-with-python-convert-model>`__
(``openvino.convert_model`` and ``openvino.save_model``)
* the `ovc <#convert-a-model-in-cli-ovc>`__ command line tool.
Once saved as :doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>` (a set of ``.xml`` and ``.bin`` files),
the model may be deployed with maximum performance. Because it is already optimized
for :doc:`OpenVINO inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>`, it can be read, compiled, and inferred with no additional delay.
Once saved as :doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`
(a set of ``.xml`` and ``.bin`` files), the model may be deployed with maximum performance.
Because it is already optimized for
:doc:`OpenVINO inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>`,
it can be read, compiled, and inferred with no additional delay.
.. note::
Model conversion API prior to OpenVINO 2023.1 is considered deprecated.
Existing and new projects are recommended to transition to the new
solutions, keeping in mind that they are not fully backwards compatible
with ``openvino.tools.mo.convert_model`` or the ``mo`` CLI tool.
For more details, see the :doc:`Model Conversion API Transition Guide <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`.
Model conversion API prior to OpenVINO 2023.1 is considered deprecated. Existing and new
projects are recommended to transition to the new solutions, keeping in mind that they are
not fully backwards compatible with ``openvino.tools.mo.convert_model`` or the ``mo``
CLI tool. For more details, see the
:doc:`Model Conversion API Transition Guide <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`.
For PyTorch models, `Python API <#convert-a-model-with-python-convert-model>`__ is the only conversion option.
For PyTorch models, `Python API <#convert-a-model-with-python-convert-model>`__ is the only
conversion option.
Model States
##############################################
There are three states a model in OpenVINO can be: saved on disk, loaded but not compiled (``ov.Model``) or loaded and compiled (``ov.CompiledModel``).
There are three states a model in OpenVINO can be: saved on disk, loaded but not compiled
(``ov.Model``) or loaded and compiled (``ov.CompiledModel``).
| **Saved on disk**
| A model in this state consists of one or more files that fully represent the neural network. A model can be stored in different ways. For example:
| A model in this state consists of one or more files that fully represent the neural
network. A model can be stored in different ways. For example:
| OpenVINO IR: pair of .xml and .bin files
| ONNX: .onnx file
| TensorFlow: directory with a .pb file and two subfolders or just a .pb file
@ -70,21 +79,26 @@ There are three states a model in OpenVINO can be: saved on disk, loaded but not
| PaddlePaddle: .pdmodel file
| **Loaded but not compiled**
| A model object (``ov.Model``) is created in memory either by parsing a file or converting an existing framework object. Inference cannot be done with this object yet as it is not attached to any specific device, but it allows customization such as reshaping its input, applying quantization or even adding preprocessing steps before compiling the model.
| A model object (``ov.Model``) is created in memory either by parsing a file or converting
an existing framework object. Inference cannot be done with this object yet as it is not
attached to any specific device, but it allows customization such as reshaping its input,
applying quantization or even adding preprocessing steps before compiling the model.
| **Loaded and compiled**
| This state is achieved when one or more devices are specified for a model object to run on (``ov.CompiledModel``), allowing device optimizations to be made and enabling inference.
| This state is achieved when one or more devices are specified for a model object to
run on (``ov.CompiledModel``), allowing device optimizations to be made and enabling
inference.
For more information on each function, see the :doc:`OpenVINO workflow <openvino_workflow>` page.
For more information on each function, see the :doc:`OpenVINO workflow <openvino_workflow>`
page.
Convert a Model with Python: ``convert_model``
##############################################
The Model conversion API in Python uses the ``openvino.convert_model`` function,
turning a given model into the `openvino.Model <api/ie_python_api/_autosummary/openvino.runtime.Model.html>`__
object and loading it to memory. Now it can be: saved to a drive with ``openvino.save_model``
or further :doc:`optimized with NNCF <openvino_docs_model_optimization_guide>`
prior to saving.
Model Conversion API in Python uses the ``openvino.convert_model`` function, to turn a model
into the `openvino.Model <api/ie_python_api/_autosummary/openvino.runtime.Model.html>`__
object and load it to memory. Now it can be: saved to a drive with ``openvino.save_model``
or further :doc:`optimized with NNCF <openvino_docs_model_optimization_guide>` before saving.
See how to use ``openvino.convert_model`` with models from some of the most popular
@ -247,19 +261,23 @@ public repositories:
* Compiling the model, **Option 2**, provides a convenient way to quickly switch from
framework-based code to OpenVINO-based code in your existing Python inference application.
The converted model is not saved to IR but compiled and used for inference within the same application.
The converted model is not saved to IR but compiled and used for inference within the same
application.
Before saving the model to OpenVINO IR, consider :doc:`Post-training Optimization <ptq_introduction>`
to achieve more efficient inference and smaller model size.
Before saving the model to OpenVINO IR, consider
:doc:`Post-training Optimization <ptq_introduction>` to achieve more efficient inference and
a smaller model.
Convert a Model in CLI: ``ovc``
###############################
``ovc`` is a command-line model converter, combining the ``openvino.convert_model``
and ``openvino.save_model`` functionalities, providing the exact same results, if the same set of
parameters is used for saving into OpenVINO IR. It converts files from one of the to :doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`, which can then be read, compiled,
and run by the final inference application.
and ``openvino.save_model`` functionalities, providing the exact same results, if the same set
of parameters is used for saving into OpenVINO IR. It converts files from one of the supported
formats to
:doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`,
which can then be read, compiled, and run by the final inference application.
.. note::
PyTorch models cannot be converted with ``ovc``, use ``openvino.convert_model`` instead.
@ -267,17 +285,20 @@ and run by the final inference application.
Additional Resources
####################
The following articles describe in detail how to obtain and prepare your model depending on the source model type:
The following articles describe in detail how to obtain and prepare your model depending on
the source model type:
* :doc:`Convert different model formats to the ov.Model format <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`.
* :doc:`Review all available conversion parameters <openvino_docs_OV_Converter_UG_Conversion_Options>`.
To achieve the best model inference performance and more compact OpenVINO IR representation follow:
To achieve the best model inference performance and more compact OpenVINO IR representation
follow:
* :doc:`Post-training optimization <ptq_introduction>`
* :doc:`Model inference in OpenVINO Runtime <openvino_docs_OV_UG_OV_Runtime_User_Guide>`
If you are still using the legacy conversion API (``mo`` or ``openvino.tools.mo.convert_model``), please refer to the following materials:
If you are still using the legacy conversion API (``mo`` or ``openvino.tools.mo.convert_model``),
refer to the following materials:
* :doc:`Transition from legacy mo and ov.tools.mo.convert_model <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`
* :doc:`Legacy Model Conversion API <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`

2
docs/articles_en/openvino_workflow/model_preparation/conversion_parameters.rst
View File

@ -67,7 +67,7 @@ Providing just a path to the model or model object as ``openvino.convert_model``
ovc path_to_your_model --compress_to_fp16=False
For details on how plugins handle compressed ``FP16`` models, see
:doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`.
:doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
.. note::

16
docs/articles_en/openvino_workflow/model_preparation/setting_input_shapes.rst
View File

@ -5,7 +5,9 @@ Setting Input Shapes
.. meta::
:description: Learn how to increase the efficiency of a model by providing an additional shape definition with the ``input`` parameter of ``openvino.convert_model`` and ``ovc``.
:description: Learn how to increase the efficiency of a model by providing an additional
shape definition with the ``input`` parameter of ``openvino.convert_model``
and ``ovc``.
``openvino.convert_model`` supports conversion of models with dynamic input shapes that
@ -15,14 +17,16 @@ one inference request to another, it is recommended to **set up static shapes**
Doing so at the model preparation stage, not at runtime, can be beneficial in terms of
performance and memory consumption.
For more information on changing input shapes in runtime, refer to the
For more information on changing input shapes in runtime, refer to the
:doc:`Changing input shapes <openvino_docs_OV_UG_ShapeInference>` guide.
To learn more about dynamic shapes in runtime, refer to the
To learn more about dynamic shapes in runtime, refer to the
:doc:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>` guide.
The OpenVINO Runtime API may present certain limitations in inferring models with undefined dimensions on some hardware.
See the :doc:`Feature support matrix <openvino_docs_OV_UG_Working_with_devices>` for reference.
In this case, the ``input`` parameter and the :doc:`reshape method <openvino_docs_OV_UG_ShapeInference>` can help to resolve undefined dimensions.
The OpenVINO Runtime API may present certain limitations in inferring models with undefined
dimensions on some hardware. See the :doc:`Feature support matrix <openvino_supported_devices>`
for reference. In this case, the ``input`` parameter and the
:doc:`reshape method <openvino_docs_OV_UG_ShapeInference>` can help to resolve undefined
dimensions.
For example, run model conversion for the TensorFlow MobileNet model with the single input
and specify the input shape of ``[2,300,300,3]``:

1
docs/articles_en/openvino_workflow/running_inference_with_openvino.rst
View File

@ -10,7 +10,6 @@ Running Inference with OpenVINO™
openvino_docs_OV_UG_Integrate_OV_with_your_application
openvino_docs_Runtime_Inference_Modes_Overview
openvino_docs_OV_UG_Working_with_devices
openvino_docs_OV_UG_ShapeInference
openvino_docs_OV_UG_DynamicShapes
openvino_docs_OV_UG_stateful_models_intro

111
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins.rst
View File

@ -1,111 +0,0 @@
.. {#openvino_docs_OV_UG_Working_with_devices}
Inference Device Support
========================
.. meta::
:description: The list of types of devices and corresponding plugins which
are compatible with OpenVINO Runtime and support inference
of deep learning models.
.. toctree::
:maxdepth: 1
:hidden:
openvino_docs_OV_UG_supported_plugins_CPU
openvino_docs_OV_UG_supported_plugins_GPU
openvino_docs_OV_UG_supported_plugins_NPU
openvino_docs_OV_UG_query_api
OpenVINO™ Runtime can infer deep learning models using the following device types:
* :doc:`CPU <openvino_docs_OV_UG_supported_plugins_CPU>`
* :doc:`GPU <openvino_docs_OV_UG_supported_plugins_GPU>`
* :doc:`NPU <openvino_docs_OV_UG_supported_plugins_NPU>`
* :doc:`Arm® CPU <openvino_docs_OV_UG_supported_plugins_CPU>`
For a more detailed list of hardware, see :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
.. _devicesupport-feature-support-matrix:
Feature Support Matrix
#######################################
The table below demonstrates support of key features by OpenVINO device plugins.
========================================================================================= ============================ ========== ===========
Capability CPU GPU NPU
========================================================================================= ============================ ========== ===========
:doc:`Heterogeneous execution <openvino_docs_OV_UG_Hetero_execution>` Yes Yes Partial
:doc:`Multi-device execution <openvino_docs_OV_UG_Running_on_multiple_devices>` Yes Yes Yes
:doc:`Automatic batching <openvino_docs_OV_UG_Automatic_Batching>` No Yes No
:doc:`Multi-stream execution <openvino_docs_deployment_optimization_guide_tput>` Yes (Intel® x86-64 only) Yes Yes
:doc:`Models caching <openvino_docs_OV_UG_Model_caching_overview>` Yes Partial Yes
:doc:`Dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` Yes Partial No
:doc:`Import/Export <openvino_ecosystem>` Yes No No*
:doc:`Preprocessing acceleration <openvino_docs_OV_UG_Preprocessing_Overview>` Yes Yes Partial
:doc:`Stateful models <openvino_docs_OV_UG_stateful_models_intro>` Yes No Yes
:doc:`Extensibility <openvino_docs_Extensibility_UG_Intro>` Yes Yes Partial
========================================================================================= ============================ ========== ===========
For more details on plugin-specific feature limitations, see the corresponding plugin pages.
Enumerating Available Devices
#######################################
The OpenVINO Runtime API features dedicated methods of enumerating devices and their capabilities. See the :doc:`Hello Query Device C++ Sample <openvino_sample_hello_query_device>`. This is an example output from the sample (truncated to device names only):
.. code-block:: sh
./hello_query_device
Available devices:
Device: CPU
...
Device: GPU.0
...
Device: GPU.1
A simple programmatic way to enumerate the devices and use with the multi-device is as follows:
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI2.cpp
:language: cpp
:fragment: [part2]
Beyond the typical "CPU", "GPU", and so on, when multiple instances of a device are available, the names are more qualified.
For example, this is how two GPUs can be listed (iGPU is always GPU.0):
.. code-block:: sh
...
Device: GPU.0
...
Device: GPU.1
So, the explicit configuration to use both would be "MULTI:GPU.1,GPU.0". Accordingly, the code that loops over all available devices of the "GPU" type only is as follows:
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI3.cpp
:language: cpp
:fragment: [part3]

9
docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide.rst
View File

@ -17,11 +17,11 @@ Optimize Inference
openvino_docs_OV_UG_Preprocessing_Overview
openvino_docs_deployment_optimization_guide_internals
Optimizing memory usage <openvino_docs_memory_optimization_guide>
.. meta::
:description: Improving inference performance involves model and runtime
optimizations that can be done independently. Inference
:description: Improving inference performance involves model and runtime
optimizations that can be done independently. Inference
speed depends on latency and throughput.
@ -34,7 +34,8 @@ It is also important to understand how the full-stack application would use the
For more information on this topic, see the following articles:
* :ref:`feature support by device <devicesupport-feature-support-matrix>`
* :doc:`Supported Devices <openvino_supported_devices>`
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`
* :ref:`Inputs Pre-processing with the OpenVINO <inputs_pre_processing>`
* :ref:`Async API <async_api>`
* :ref:`The 'get_tensor' Idiom <tensor_idiom>`

10
docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_latency.rst
View File

@ -11,8 +11,8 @@ Optimizing for Latency
openvino_docs_OV_UG_Model_caching_overview
.. meta::
:description: OpenVINO provides methods that help to preserve minimal
latency despite the number of inference requests and
:description: OpenVINO provides methods that help to preserve minimal
latency despite the number of inference requests and
improve throughput without degrading latency.
@ -34,15 +34,15 @@ Typically, human expertise is required to get more "throughput" out of the devic
* feature support by device
When multiple models are to be used simultaneously, consider running inference on separate devices for each of them. Finally, when multiple models are executed in parallel on a device, using additional ``ov::hint::model_priority`` may help to define relative priorities of the models. Refer to the documentation on the :ref:`OpenVINO feature support for devices <devicesupport-feature-support-matrix>` to check if your device supports the feature.
When multiple models are to be used simultaneously, consider running inference on separate devices for each of them. Finally, when multiple models are executed in parallel on a device, using additional ``ov::hint::model_priority`` may help to define relative priorities of the models. Refer to the documentation on the :ref:`OpenVINO feature support for devices <openvino_supported_devices>` to check if your device supports the feature.
**First-Inference Latency and Model Load/Compile Time**
In some cases, model loading and compilation contribute to the "end-to-end" latency more than usual.
In some cases, model loading and compilation contribute to the "end-to-end" latency more than usual.
For example, when the model is used exactly once, or when it is unloaded and reloaded in a cycle, to free the memory for another inference due to on-device memory limitations.
Such a "first-inference latency" scenario may pose an additional limitation on the model load\compilation time, as inference accelerators (other than the CPU) usually require a certain level of model compilation upon loading.
The :doc:`model caching <openvino_docs_OV_UG_Model_caching_overview>` option is a way to lessen the impact over multiple application runs. If model caching is not possible, for example, it may require write permissions for the application, the CPU offers the fastest model load time almost every time.
The :doc:`model caching <openvino_docs_OV_UG_Model_caching_overview>` option is a way to lessen the impact over multiple application runs. If model caching is not possible, for example, it may require write permissions for the application, the CPU offers the fastest model load time almost every time.
To improve common "first-inference latency" scenario, model reading was replaced with model mapping (using `mmap`) into a memory. But in some use cases (first of all, if model is located on removable or network drive) mapping may lead to latency increase. To switch mapping to reading, specify ``ov::enable_mmap(false)`` property for the ``ov::Core``.

43
docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/performance_hints.rst
View File

@ -5,13 +5,13 @@ High-level Performance Hints
.. meta::
:description: OpenVINO Runtime offers two dedicated high-level performance
hints, namely throughput and latency, that help to configure
:description: OpenVINO Runtime offers two dedicated high-level performance
hints, namely throughput and latency, that help to configure
an inference device.
Even though all :doc:`supported devices <openvino_docs_OV_UG_Working_with_devices>` in OpenVINO™ offer low-level performance settings, utilizing them is not recommended outside of very few cases.
The preferred way to configure performance in OpenVINO Runtime is using performance hints. This is a future-proof solution fully compatible with the :doc:`automatic device selection inference mode <openvino_docs_OV_UG_supported_plugins_AUTO>` and designed with *portability* in mind.
Even though all :doc:`supported devices <openvino_supported_devices>` in OpenVINO™ offer low-level performance settings, utilizing them is not recommended outside of very few cases.
The preferred way to configure performance in OpenVINO Runtime is using performance hints. This is a future-proof solution fully compatible with the :doc:`automatic device selection inference mode <openvino_docs_OV_UG_supported_plugins_AUTO>` and designed with *portability* in mind.
The hints also set the direction of the configuration in the right order. Instead of mapping the application needs to the low-level performance settings, and keeping an associated application logic to configure each possible device separately, the hints express a target scenario with a single config key and let the *device* configure itself in response.
@ -34,7 +34,7 @@ Performance Hints: How It Works
Internally, every device "translates" the value of the hint to the actual performance settings.
For example, the ``ov::hint::PerformanceMode::THROUGHPUT`` selects the number of CPU or GPU streams.
Additionally, the optimal batch size is selected for the GPU and the :doc:`automatic batching <openvino_docs_OV_UG_Automatic_Batching>` is applied whenever possible. To check whether the device supports it, refer to the :doc:`devices/features support matrix <openvino_docs_OV_UG_Working_with_devices>` article.
Additionally, the optimal batch size is selected for the GPU and the :doc:`automatic batching <openvino_docs_OV_UG_Automatic_Batching>` is applied whenever possible. To check whether the device supports it, refer to the :doc:`Supported devices <openvino_supported_devices>` article.
The resulting (device-specific) settings can be queried back from the instance of the ``ov:Compiled_Model``.
Be aware that the ``benchmark_app`` outputs the actual settings for the ``THROUGHPUT`` hint. See the example of the output below:
@ -61,14 +61,14 @@ In the example code snippet below, ``ov::hint::PerformanceMode::THROUGHPUT`` is
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: python
:fragment: [compile_model]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [compile_model]
@ -77,21 +77,21 @@ In the example code snippet below, ``ov::hint::PerformanceMode::THROUGHPUT`` is
Additional (Optional) Hints from the App
########################################
For an application that processes 4 video streams, the most future-proof way to communicate the limitation of the parallel slack is to equip the performance hint with the optional ``ov::hint::num_requests`` configuration key set to 4.
For an application that processes 4 video streams, the most future-proof way to communicate the limitation of the parallel slack is to equip the performance hint with the optional ``ov::hint::num_requests`` configuration key set to 4.
As mentioned earlier, this will limit the batch size for the GPU and the number of inference streams for the CPU. Thus, each device uses the ``ov::hint::num_requests`` while converting the hint to the actual device configuration options:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: python
:fragment: [hint_num_requests]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [hint_num_requests]
@ -106,20 +106,20 @@ The hints are used on the presumption that the application queries ``ov::optimal
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: python
:fragment: [query_optimal_num_requests]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [query_optimal_num_requests]
While an application is free to create more requests if needed (for example to support asynchronous inputs population) **it is very important to at least run the** ``ov::optimal_number_of_infer_requests`` **of the inference requests in parallel**. It is recommended for efficiency, or device utilization, reasons.
While an application is free to create more requests if needed (for example to support asynchronous inputs population) **it is very important to at least run the** ``ov::optimal_number_of_infer_requests`` **of the inference requests in parallel**. It is recommended for efficiency, or device utilization, reasons.
Keep in mind that ``ov::hint::PerformanceMode::LATENCY`` does not necessarily imply using single inference request. For example, multi-socket CPUs can deliver as many requests at the same minimal latency as the number of NUMA nodes in the system.
To make your application fully scalable, make sure to query the ``ov::optimal_number_of_infer_requests`` directly.
@ -131,30 +131,30 @@ Prefer Async API
The API of the inference requests offers Sync and Async execution. The ``ov::InferRequest::infer()`` is inherently synchronous and simple to operate (as it serializes the execution flow in the current application thread). The Async "splits" the ``infer()`` into ``ov::InferRequest::start_async()`` and ``ov::InferRequest::wait()`` (or callbacks). For more information on synchronous and asynchronous modes, refer to the :doc:`OpenVINO Inference Request documentation <openvino_docs_OV_UG_Infer_request>`.
Although the synchronous API can be easier to start with, it is recommended to use the asynchronous (callbacks-based) API in production code. It is the most general and scalable way to implement the flow control for any possible number of requests. The ``THROUGHPUT`` and ``LATENCY`` performance hints automatically configure the Asynchronous pipeline to use the optimal number of processing streams and inference requests.
Although the synchronous API can be easier to start with, it is recommended to use the asynchronous (callbacks-based) API in production code. It is the most general and scalable way to implement the flow control for any possible number of requests. The ``THROUGHPUT`` and ``LATENCY`` performance hints automatically configure the Asynchronous pipeline to use the optimal number of processing streams and inference requests.
.. note::
**Important:** Performance Hints only work when asynchronous execution mode is used. They do not affect the performance of a synchronous pipeline.
Combining the Hints and Individual Low-Level Settings
#####################################################
While sacrificing the portability to some extent, it is possible to combine the hints with individual device-specific settings.
While sacrificing the portability to some extent, it is possible to combine the hints with individual device-specific settings.
For example, use ``ov::hint::PerformanceMode::THROUGHPUT`` to prepare a general configuration and override any of its specific values:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: python
:fragment: [hint_plus_low_level]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [hint_plus_low_level]
@ -171,10 +171,3 @@ Using the :doc:`benchmark_app sample <openvino_sample_benchmark_tool>`is the bes
Disabling the hints to emulate the pre-hints era (highly recommended before trying the individual low-level settings, such as the number of streams as below, threads, etc):
* benchmark_app **-hint none -nstreams 1** -d 'device' -m 'path to your model'
Additional Resources
####################
* :doc:`Supported Devices <openvino_docs_OV_UG_Working_with_devices>`

2
docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/precision_control.rst
View File

@ -59,6 +59,6 @@ Inference Precision
Additional Resources
####################
* :doc:`Supported Devices <openvino_docs_OV_UG_Working_with_devices>`
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`

86
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes.rst Normal file
View File

@ -0,0 +1,86 @@
.. {#openvino_docs_Runtime_Inference_Modes_Overview}
Inference Devices and Modes
============================
.. toctree::
:maxdepth: 1
:hidden:
openvino_docs_OV_UG_supported_plugins_CPU
openvino_docs_OV_UG_supported_plugins_GPU
openvino_docs_OV_UG_supported_plugins_NPU
openvino_docs_OV_UG_supported_plugins_AUTO
openvino_docs_OV_UG_Running_on_multiple_devices
openvino_docs_OV_UG_Hetero_execution
openvino_docs_OV_UG_Automatic_Batching
openvino_docs_OV_UG_query_api
The OpenVINO runtime offers multiple inference modes to enable the best hardware utilization under
different conditions:
| **single-device inference**
| Define just one device responsible for the entire inference workload. It supports a range of
processors by means of the following plugins embedded in the Runtime library:
| - :doc:`CPU <openvino_docs_OV_UG_supported_plugins_CPU>`
| - :doc:`GPU <openvino_docs_OV_UG_supported_plugins_GPU>`
| - :doc:`NPU <openvino_docs_OV_UG_supported_plugins_NPU>`
| **automated inference modes**
| Assume certain level of automation in selecting devices for inference. They may potentially
increase your deployed solution's performance and portability. The automated modes are:
| - :doc:`Automatic Device Selection (AUTO) <openvino_docs_OV_UG_supported_plugins_AUTO>`
| - :doc:`Multi-Device Execution (MULTI) <openvino_docs_OV_UG_Running_on_multiple_devices>`
| - :doc:`Heterogeneous Execution (HETERO) <openvino_docs_OV_UG_Hetero_execution>`
| - :doc:`Automatic Batching Execution (Auto-batching) <openvino_docs_OV_UG_Automatic_Batching>`
Enumerating Available Devices
#######################################
The OpenVINO Runtime API features dedicated methods of enumerating devices and their capabilities.
Note that beyond the typical "CPU" or "GPU" device names, more qualified names are used when multiple
instances of a device are available (iGPU is always GPU.0).
The output you receive may look like this (truncated to device names only, two GPUs are listed
as an example):
.. code-block:: sh
./hello_query_device
Available devices:
Device: CPU
...
Device: GPU.0
...
Device: GPU.1
You may see how to obtain this information in the :doc:`Hello Query Device Sample <openvino_sample_hello_query_device>`.
Here is an example of a simple programmatic way to enumerate the devices and use them with the
multi-device mode:
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI2.cpp
:language: cpp
:fragment: [part2]
With two GPU devices used in one setup, the explicit configuration would be "MULTI:GPU.1,GPU.0".
Accordingly, the code that loops over all available devices of the "GPU" type only is as follows:
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI3.cpp
:language: cpp
:fragment: [part3]

2
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/CPU.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/CPU.rst
View File

@ -543,7 +543,7 @@ Currently, the ``sparse weights decompression feature`` is supported with the fo
Additional Resources
###########################################################
* :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`
* :doc:`Optimization guide <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`
* `CPU plugin developers documentation <https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_cpu/README.md>`__

2
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/GPU.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/GPU.rst
View File

@ -476,7 +476,7 @@ Since OpenVINO relies on the OpenCL kernels for the GPU implementation, many gen
Additional Resources
#######################################
* :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
* :doc:`Optimization guide <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
* `GPU plugin developers documentation <https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_gpu/README.md>`__

0
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/GPU/GPU_RemoteTensor_API.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/GPU/GPU_RemoteTensor_API.rst
View File

0
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/NPU.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/NPU.rst
View File

5
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview/auto_device_selection.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/auto_device_selection.rst
View File

@ -247,7 +247,8 @@ See the following code for using AUTO and specifying devices:
Note that OpenVINO Runtime lets you use "GPU" as an alias for "GPU.0" in function calls. More details on enumerating devices can be found in :doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`.
Note that OpenVINO Runtime lets you use "GPU" as an alias for "GPU.0" in function calls.
More details on enumerating devices can be found in :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
Checking Available Devices
--------------------------
@ -508,6 +509,6 @@ Additional Resources
- :doc:`Debugging AUTO <openvino_docs_OV_UG_supported_plugins_AUTO_debugging>`
- :doc:`Running on Multiple Devices Simultaneously <openvino_docs_OV_UG_Running_on_multiple_devices>`
- :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`
- :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`

0
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview/auto_device_selection/AutoPlugin_Debugging.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/auto_device_selection/AutoPlugin_Debugging.rst
View File

60
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview/automatic_batching.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/automatic_batching.rst
View File

@ -5,18 +5,18 @@ Automatic Batching
.. meta::
:description: The Automatic Batching Execution mode in OpenVINO Runtime
performs automatic batching to improve device utilization
:description: The Automatic Batching Execution mode in OpenVINO Runtime
performs automatic batching to improve device utilization
by grouping inference requests.
The Automatic Batching Execution mode (or Auto-batching for short) performs automatic batching on-the-fly to improve device utilization by grouping inference requests together, without programming effort from the user.
With Automatic Batching, gathering the input and scattering the output from the individual inference requests required for the batch happen transparently, without affecting the application code.
With Automatic Batching, gathering the input and scattering the output from the individual inference requests required for the batch happen transparently, without affecting the application code.
Auto Batching can be used :ref:`directly as a virtual device <auto-batching-as-device>` or as an :ref:`option for inference on CPU/GPU/NPU <auto-batching-as-option>` (by means of configuration/hint). These 2 ways are provided for the user to enable the BATCH devices **explicitly** or **implicitly**, with the underlying logic remaining the same. An example of the difference is that the CPU device doesn’t support implicitly to enable BATCH device, commands such as ``./benchmark_app -m <model> -d CPU -hint tput`` will not apply BATCH device **implicitly**, but ``./benchmark_app -m <model> -d "BATCH:CPU(16)`` can **explicitly** load BATCH device.
Auto-batching primarily targets the existing code written for inferencing many requests, each instance with the batch size 1. To get corresponding performance improvements, the application **must be running multiple inference requests simultaneously**.
Auto-batching can also be used via a particular *virtual* device.
Auto-batching primarily targets the existing code written for inferencing many requests, each instance with the batch size 1. To get corresponding performance improvements, the application **must be running multiple inference requests simultaneously**.
Auto-batching can also be used via a particular *virtual* device.
This article provides a preview of the Automatic Batching function, including how it works, its configurations, and testing performance.
@ -24,28 +24,28 @@ How Automatic Batching Works
############################
.. tab-set::
.. tab-item:: Enabling Automatic Batching
:sync: enabling-automatic-batching
Batching is a straightforward way of leveraging the compute power of GPU and saving on communication overheads. Automatic Batching is "implicitly" triggered on the GPU when ``ov::hint::PerformanceMode::THROUGHPUT`` is specified for the ``ov::hint::performance_mode`` property for the ``compile_model`` or ``set_property`` calls.
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: Python
:fragment: [compile_model]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [compile_model]
To enable Auto-batching in the legacy apps not akin to the notion of performance hints, you need to use the **explicit** device notion, such as ``BATCH:GPU``.
.. tab-item:: Disabling Automatic Batching
@ -54,17 +54,17 @@ How Automatic Batching Works
Auto-Batching can be disabled (for example, for the GPU device) to prevent being triggered by ``ov::hint::PerformanceMode::THROUGHPUT``. To do that, set ``ov::hint::allow_auto_batching`` to **false** in addition to the ``ov::hint::performance_mode``, as shown below:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: Python
:fragment: [compile_model_no_auto_batching]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [compile_model_no_auto_batching]
@ -99,10 +99,10 @@ This "automatic batch size selection" works on the presumption that the applicat
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: Python
:fragment: [query_optimal_num_requests]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [query_optimal_num_requests]
@ -127,10 +127,10 @@ For example, when the application processes only 4 video streams, there is no ne
.. doxygensnippet:: docs/snippets/ov_auto_batching.py
:language: Python
:fragment: [hint_num_requests]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_auto_batching.cpp
:language: cpp
:fragment: [hint_num_requests]
@ -153,7 +153,7 @@ The below examples show how AUTO Batching can be used in the form of device that
./benchmark_app -m <model> -d "BATCH:CPU(16)"
* ``BATCH`` -- load BATCH device explicitly,
* ``BATCH`` -- load BATCH device explicitly,
* ``:GPU(16)`` -- BATCH devices configuration, which tell BATCH device to apply GPU device with batch size = 16.
.. _auto-batching-as-option:
@ -167,7 +167,7 @@ In the following example, BATCH device will be configured to another device in c
./benchmark_app -m <model> -d GPU -hint tput
./benchmark_app -m <model> -d AUTO -hint tput
./benchmark_app -m <model> -d AUTO -hint ctput
./benchmark_app -m <model> -d AUTO -hint ctput
./benchmark_app -m <model> -d AUTO:GPU -hint ctput
.. note::
@ -199,7 +199,7 @@ The following are limitations of the current AUTO Batching implementations:
- Although it is less critical for the throughput-oriented scenarios, the load time with Auto-batching increases by almost double.
- Certain networks are not safely reshapable by the "batching" dimension (specified as ``N`` in the layout terms). Besides, if the batching dimension is not zeroth, Auto-batching will not be triggered "implicitly" by the throughput hint.
- The "explicit" notion, for example, ``BATCH:GPU``, using the relaxed dimensions tracking, often makes Auto-batching possible. For example, this method unlocks most **detection networks**.
- When *forcing* Auto-batching via the "explicit" device notion, make sure that you validate the results for correctness.
- When *forcing* Auto-batching via the "explicit" device notion, make sure that you validate the results for correctness.
- Performance improvements happen at the cost of the growth of memory footprint. However, Auto-batching queries the available memory (especially for dGPU) and limits the selected batch size accordingly.
@ -209,13 +209,13 @@ Testing Performance with Benchmark_app
Using the :doc:`benchmark_app sample <openvino_sample_benchmark_tool>` is the best way to evaluate the performance of Automatic Batching:
- The most straightforward way is using the performance hints:
- benchmark_app **-hint tput** -d GPU -m 'path to your favorite model'
- You can also use the "explicit" device notion to override the strict rules of the implicit reshaping by the batch dimension:
- benchmark_app **-hint none -d BATCH:GPU** -m 'path to your favorite model'
- or override the automatically deduced batch size as well:
- $benchmark_app -hint none -d **BATCH:GPU(16)** -m 'path to your favorite model'
- This example also applies to CPU or any other device that generally supports batch execution.
- Keep in mind that some shell versions (e.g. ``bash``) may require adding quotes around complex device names, i.e. ``-d "BATCH:GPU(16)"`` in this example.
@ -225,21 +225,21 @@ Note that Benchmark_app performs a warm-up run of a *single* request. As Auto-Ba
.. code-block:: sh
[ INFO ] First inference took 1000.18ms
[ INFO ] First inference took 1000.18ms
This value also exposed as the final execution statistics on the ``benchmark_app`` exit:
.. code-block:: sh
[ INFO ] Latency:
[ INFO ] Latency:
[ INFO ] Max: 1000.18 ms
This is NOT the actual latency of the batched execution, so you are recommended to refer to other metrics in the same log, for example, "Median" or "Average" execution.
This is NOT the actual latency of the batched execution, so you are recommended to refer to other metrics in the same log, for example, "Median" or "Average" execution.
Additional Resources
####################
* :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`

0
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/config_properties.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/config_properties.rst
View File

2
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview/hetero_execution.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/hetero_execution.rst
View File

@ -204,5 +204,5 @@ You can also point to more than two devices: ``-d HETERO:GPU,CPU``
Additional Resources
####################
* :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`
* :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`

18
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview/multi_device.rst → docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_devices_and_modes/multi_device.rst
View File

@ -5,8 +5,8 @@ Multi-device execution
.. meta::
:description: The Multi-Device execution mode in OpenVINO Runtime assigns
multiple available computing devices to particular inference
:description: The Multi-Device execution mode in OpenVINO Runtime assigns
multiple available computing devices to particular inference
requests to execute in parallel.
@ -49,7 +49,7 @@ Following the OpenVINO™ naming convention, the Multi-Device mode is assigned t
Specifying the device list explicitly is required by MULTI, as it defines the devices available for inference and sets their priorities.
Note that OpenVINO™ Runtime enables you to use “GPU” as an alias for “GPU.0” in function calls. More details on enumerating devices can be found in :doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`.
Note that OpenVINO™ Runtime enables you to use “GPU” as an alias for “GPU.0” in function calls. More details on enumerating devices can be found in :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`.
The following commands are accepted by the API:
@ -57,14 +57,14 @@ The following commands are accepted by the API:
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_multi.py
:language: python
:fragment: [MULTI_0]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI0.cpp
:language: cpp
:fragment: [part0]
@ -82,14 +82,14 @@ As mentioned previously, executing inference with MULTI may be set up by configu
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_multi.py
:language: python
:fragment: [MULTI_4]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/MULTI4.cpp
:language: cpp
:fragment: [part4]
@ -106,7 +106,7 @@ using the :doc:`configure devices <openvino_docs_OV_UG_query_api>` property:
.. tab-set::
.. tab-item:: C++
.. doxygensnippet:: docs/snippets/MULTI5.cpp
:language: cpp
:fragment: [part5]
@ -148,7 +148,7 @@ For best performance when using the MULTI execution mode you should consider a f
Additional Resources
####################
- :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`
- :doc:`Inference Devices and Modes <openvino_docs_Runtime_Inference_Modes_Overview>`
- :doc:`Automatic Device Selection <openvino_docs_OV_UG_supported_plugins_AUTO>`

25
docs/articles_en/openvino_workflow/running_inference_with_openvino/inference_modes_overview.rst
View File

@ -1,25 +0,0 @@
.. {#openvino_docs_Runtime_Inference_Modes_Overview}
Inference Modes
===============
.. toctree::
:maxdepth: 1
:hidden:
openvino_docs_OV_UG_supported_plugins_AUTO
openvino_docs_OV_UG_Running_on_multiple_devices
openvino_docs_OV_UG_Hetero_execution
openvino_docs_OV_UG_Automatic_Batching
OpenVINO Runtime offers multiple inference modes to allow optimum hardware utilization under different conditions. The most basic one is a single-device mode, which defines just one device responsible for the entire inference workload. It supports a range of Intel hardware by means of plugins embedded in the Runtime library, each set up to offer the best possible performance. For a complete list of supported devices and instructions on how to use them, refer to the :doc:`guide on inference devices <openvino_docs_OV_UG_Working_with_devices>`.
The remaining modes assume certain levels of automation in selecting devices for inference. Using them in the deployed solution may potentially increase its performance and portability. The automated modes are:
* :doc:`Automatic Device Selection (AUTO) <openvino_docs_OV_UG_supported_plugins_AUTO>`
* :doc:`Multi-Device Execution (MULTI) <openvino_docs_OV_UG_Running_on_multiple_devices>`
* :doc:`Heterogeneous Execution (HETERO) <openvino_docs_OV_UG_Hetero_execution>`
* :doc:`Automatic Batching Execution (Auto-batching) <openvino_docs_OV_UG_Automatic_Batching>`

30
docs/articles_en/openvino_workflow/running_inference_with_openvino/ov_dynamic_shapes.rst
View File

@ -11,8 +11,8 @@ Dynamic Shapes
openvino_docs_OV_UG_NoDynamicShapes
.. meta::
:description: The Reshape method in OpenVINO Runtime API can handle dynamic
shapes of models that support changing input shapes before
:description: The Reshape method in OpenVINO Runtime API can handle dynamic
shapes of models that support changing input shapes before
model compilation.
@ -61,7 +61,7 @@ Configuring the Model
Model input dimensions can be specified as dynamic using the model.reshape method. To set a dynamic dimension, use ``-1``, ``ov::Dimension()`` (C++), or ``ov.Dimension()`` (Python) as the value for that dimension.
.. note::
Some models may already have dynamic shapes out of the box and do not require additional configuration. This can either be because it was generated with dynamic shapes from the source framework, or because it was converted with Model Conversion API to use dynamic shapes. For more information, see the Dynamic Dimensions “Out of the Box” section.
The examples below show how to set dynamic dimensions with a model that has a static ``[1, 3, 224, 224]`` input shape (such as `mobilenet-v2 <https://docs.openvino.ai/2023.3/omz_models_model_mobilenet_v2.html>`__). The first example shows how to change the first dimension (batch size) to be dynamic. In the second example, the third and fourth dimensions (height and width) are set as dynamic.
@ -74,7 +74,7 @@ The examples below show how to set dynamic dimensions with a model that has a st
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: cpp
:fragment: [reshape_undefined]
With Python, you may also pass all dimensions as a string and use ``?`` for the dynamic dimensions (e.g. ``model.reshape(“1, 3, ?, ?”)``).
.. tab-item:: C++
@ -146,37 +146,37 @@ Static and dynamic dimensions can also be set when converting the model with ``c
Dimension Bounds
----------------
The lower and/or upper bounds of a dynamic dimension can also be specified. They define a range of allowed values for the dimension. Dimension bounds can be set by passing the lower and upper bounds into the ``reshape`` method using the options shown below.
The lower and/or upper bounds of a dynamic dimension can also be specified. They define a range of allowed values for the dimension. Dimension bounds can be set by passing the lower and upper bounds into the ``reshape`` method using the options shown below.
.. tab-set::
.. tab-item:: Python
:sync: py
Each of these options are equivalent:
- Pass the lower and upper bounds directly into the ``reshape`` method, e.g. ``model.reshape([1, 10), (8,512)])``
- Pass the lower and upper bounds using ov.Dimension, e.g. ``model.reshape([ov.Dimension(1, 10), (8, 512)])``
- Pass the dimension ranges as strings, e.g. ``model.reshape(“1..10, 8..512”)``
The examples below show how to set dynamic dimension bounds for a mobilenet-v2 model with a default static shape of ``[1,3,224,224]``.
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: reshape_bounds
.. tab-item:: C++
:sync: cpp
The dimension bounds can be coded as arguments for ``ov::Dimension``, as shown in these examples:
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: ov_dynamic_shapes:reshape_bounds
.. tab-item:: C
:sync: c
The dimension bounds can be coded as arguments for `ov_dimension <https://docs.openvino.ai/2023.3/api/c_cpp_api/structov__dimension.html>`__, as shown in these examples:
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.c
@ -192,7 +192,7 @@ For the same reason, it is not recommended to leave dimensions as undefined, wit
When specifying bounds, the lower bound is not as important as the upper one. The upper bound allows inference devices to allocate memory for intermediate tensors more precisely. It also allows using a fewer number of tuned kernels for different sizes.
More precisely, benefits of specifying the lower or upper bound is device dependent.
Depending on the plugin, specifying the upper bounds can be required. For information about dynamic shapes support on different devices, refer to the :ref:`feature support table <devicesupport-feature-support-matrix>`.
Depending on the plugin, specifying the upper bounds can be required. For information about dynamic shapes support on different devices, refer to the :doc:`feature support table <openvino_supported_devices>`.
If the lower and upper bounds for a dimension are known, it is recommended to specify them, even if a plugin can execute a model without the bounds.
@ -218,7 +218,7 @@ The sample below shows how a model can accept different input shapes. In the fir
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: ov_dynamic_shapes:set_input_tensor
.. tab-item:: C
:sync: c