[DOCS] Recreation of BDTI PRs - 22.3 (#17546)
* 16913 * Update performance_hints.md
This commit is contained in:
Maciej Smyk
GitHub
parent
7303ffa7b9
commit
f392465e2a
|
|
@ -66,12 +66,12 @@ Note that if you choose to exclude CPU from the priority list or disable the ini
|
|||
This mechanism can be easily observed in the :ref:`Using AUTO with Benchmark app sample <using-auto-with-openvino-samples-and-benchmark-app>` section, showing how the first-inference latency (the time it takes to compile the model and perform the first inference) is reduced when using AUTO. For example:
|
||||
|
||||
|
||||
.. code-block: sh
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m ../public/alexnet/FP32/alexnet.xml -d GPU -niter 128
|
||||
|
||||
|
||||
.. code-block: sh
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m ../public/alexnet/FP32/alexnet.xml -d AUTO -niter 128
|
||||
|
||||
|
|
@ -224,9 +224,14 @@ Performance Hints for AUTO
|
|||
|
||||
The ``ov::hint::performance_mode`` property enables you to specify a performance option for AUTO to be more efficient for particular use cases. The default hint for AUTO is ``LATENCY``.
|
||||
|
||||
The THROUGHPUT and CUMULATIVE_THROUGHPUT hints below only improve performance in an asynchronous inference pipeline. For information on asynchronous inference, see the `Async API documentation <https://docs.openvino.ai/2022.3/openvino_docs_OV_UG_Infer_request.html#doxid-openvino-docs-o-v-u-g-infer-request>`__ . The following notebooks provide examples of how to set up an asynchronous pipeline:
|
||||
|
||||
* :doc:`Image Classification Async Sample <openvino_inference_engine_samples_classification_sample_async_README>`
|
||||
* `Notebook - Asynchronous Inference with OpenVINO™ <https://docs.openvino.ai/2022.3/notebooks/115-async-api-with-output.html>`__
|
||||
* `Notebook - Automatic Device Selection with OpenVINO <https://docs.openvino.ai/2022.3/notebooks/106-auto-device-with-output.html>`__
|
||||
|
||||
LATENCY
|
||||
^^^^^^^
|
||||
----------
|
||||
|
||||
This option prioritizes low latency, providing short response time for each inference job. It performs best for tasks where inference is required for a single input image, e.g. a medical analysis of an ultrasound scan image. It also fits the tasks of real-time or nearly real-time applications, such as an industrial robot's response to actions in its environment or obstacle avoidance for autonomous vehicles.
|
||||
|
||||
|
|
@ -251,15 +256,43 @@ While ``LATENCY`` and ``THROUGHPUT`` can select one target device with your pref
|
|||
|
||||
CUMULATIVE_THROUGHPUT has similar behavior as :doc:`the Multi-Device execution mode (MULTI) <openvino_docs_OV_UG_Running_on_multiple_devices>`. The only difference is that CUMULATIVE_THROUGHPUT uses the devices specified by AUTO, which means that it's not mandatory to add devices manually, while with MULTI, you need to specify the devices before inference.
|
||||
|
||||
With the CUMULATIVE_THROUGHPUT option:
|
||||
If device priority is specified when using CUMULATIVE_THROUGHPUT, AUTO will run inference requests on devices based on the priority. In the following example, AUTO will always try to use GPU first, and then use CPU if GPU is busy:
|
||||
|
||||
* If ``AUTO`` without any device names is specified, and the system has more than two GPU devices, AUTO will remove CPU from the device candidate list to keep GPU running at full capacity.
|
||||
* If device priority is specified, AUTO will run inference requests on devices based on the priority. In the following example, AUTO will always try to use GPU first, and then use CPU if GPU is busy:
|
||||
.. tab-set::
|
||||
|
||||
.. code-block: sh
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU,CPU", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT));
|
||||
.. code-block:: sh
|
||||
|
||||
ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU,CPU", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT));
|
||||
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
compiled_model = core.compile_model(model, "AUTO:GPU,CPU", {"PERFORMANCE_HINT" : {"CUMULATIVE_THROUGHPUT"}})
|
||||
|
||||
|
||||
If AUTO is used without specifying any device names, and if there are multiple GPUs in the system, CUMULATIVE_THROUGHPUT mode will use all of the GPUs by default. If the system has more than two GPU devices, AUTO will remove CPU from the device candidate list to keep the GPUs running at full capacity. A full list of system devices and their unique identifiers can be queried using ov::Core::get_available_devices (for more information, see :doc:`Query Device Properties <openvino_docs_OV_UG_query_api>`). To explicitly specify which GPUs to use, set their priority when compiling with AUTO:
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU.1,GPU.0", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT));
|
||||
|
||||
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
compiled_model = core.compile_model(model, "AUTO:GPU.1,GPU.0", {"PERFORMANCE_HINT" : {"CUMULATIVE_THROUGHPUT"})
|
||||
|
||||
Code Examples
|
||||
--------------------
|
||||
|
|
@ -267,17 +300,21 @@ Code Examples
|
|||
To enable performance hints for your application, use the following code:
|
||||
|
||||
|
||||
.. tab:: C++
|
||||
.. tab-set::
|
||||
|
||||
.. doxygensnippet:: docs/snippets/AUTO3.cpp
|
||||
:language: cpp
|
||||
:fragment: [part3]
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
.. tab:: Python
|
||||
.. doxygensnippet:: docs/snippets/AUTO3.cpp
|
||||
:language: cpp
|
||||
:fragment: [part3]
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part3]
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part3]
|
||||
|
||||
|
||||
Disabling Auto-Batching for THROUGHPUT and CUMULATIVE_THROUGHPUT
|
||||
|
|
@ -292,17 +329,21 @@ Configuring Model Priority
|
|||
The ``ov::hint::model_priority`` property enables you to control the priorities of models in the Auto-Device plugin. A high-priority model will be loaded to a supported high-priority device. A lower-priority model will not be loaded to a device that is occupied by a higher-priority model.
|
||||
|
||||
|
||||
.. tab:: C++
|
||||
.. tab-set::
|
||||
|
||||
.. doxygensnippet:: docs/snippets/AUTO4.cpp
|
||||
:language: cpp
|
||||
:fragment: [part4]
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
.. tab:: Python
|
||||
.. doxygensnippet:: docs/snippets/AUTO4.cpp
|
||||
:language: cpp
|
||||
:fragment: [part4]
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part4]
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part4]
|
||||
|
||||
|
||||
Checking Target Runtime Devices
|
||||
|
|
@ -311,35 +352,43 @@ Checking Target Runtime Devices
|
|||
To query the runtime target devices on which the inferences are being executed using AUTO, you can use the ``ov::execution_devices`` property. It must be used with ``get_property``, for example:
|
||||
|
||||
|
||||
.. tab:: C++
|
||||
.. tab-set::
|
||||
|
||||
.. doxygensnippet:: docs/snippets/AUTO7.cpp
|
||||
:language: cpp
|
||||
:fragment: [part7]
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
.. tab:: Python
|
||||
.. doxygensnippet:: docs/snippets/AUTO7.cpp
|
||||
:language: cpp
|
||||
:fragment: [part7]
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part7]
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part7]
|
||||
|
||||
|
||||
Configuring Individual Devices and Creating the Auto-Device plugin on Top
|
||||
#########################################################################
|
||||
|
||||
Although the methods described above are currently the preferred way to execute inference with AUTO, the following steps can be also used as an alternative. It is currently available as a legacy feature and used if the device candidate list includes Myriad devices, incapable of utilizing the Performance Hints option.
|
||||
Although the methods described above are currently the preferred way to execute inference with AUTO, the following steps can be also used as an alternative. It is currently available as a legacy feature and used if AUTO is incapable of utilizing the Performance Hints option.
|
||||
|
||||
.. tab:: C++
|
||||
.. tab-set::
|
||||
|
||||
.. doxygensnippet:: docs/snippets/AUTO5.cpp
|
||||
:language: cpp
|
||||
:fragment: [part5]
|
||||
.. tab-item:: C++
|
||||
:sync: cpp
|
||||
|
||||
.. tab:: Python
|
||||
.. doxygensnippet:: docs/snippets/AUTO5.cpp
|
||||
:language: cpp
|
||||
:fragment: [part5]
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part5]
|
||||
.. tab-item:: Python
|
||||
:sync: py
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_auto.py
|
||||
:language: python
|
||||
:fragment: [part5]
|
||||
|
||||
|
||||
.. _using-auto-with-openvino-samples-and-benchmark-app:
|
||||
|
|
@ -351,15 +400,15 @@ To see how the Auto-Device plugin is used in practice and test its performance,
|
|||
|
||||
For unlimited device choice:
|
||||
|
||||
.. code-block:sh
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app –d AUTO –m <model> -i <input> -niter 1000
|
||||
|
||||
For limited device choice:
|
||||
|
||||
.. code-block:sh
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app –d AUTO:CPU,GPU,MYRIAD –m <model> -i <input> -niter 1000
|
||||
benchmark_app –d AUTO:CPU,GPU,GNA –m <model> -i <input> -niter 1000
|
||||
|
||||
For more information, refer to the :doc:`C++ <openvino_inference_engine_samples_benchmark_app_README>` or :doc:`Python <openvino_inference_engine_tools_benchmark_tool_README>` version instructions.
|
||||
|
||||
|
|
|
|||
|
|
@ -38,19 +38,21 @@ The decision about using dynamic shapes should be based on proper benchmarking o
|
|||
Unlike statically shaped models, dynamically shaped ones require different inference time, depending on input data shape or input tensor content.
|
||||
Furthermore, using the dynamic shapes can bring more overheads in memory and running time of each inference call depending on hardware plugin and model used.
|
||||
|
||||
## Handling Dynamic Shapes Natively
|
||||
## Handling Dynamic Shapes
|
||||
|
||||
This section describes how to handle dynamically shaped models natively with OpenVINO Runtime API version 2022.1 and higher.
|
||||
There are three main parts in the flow that differ from static shapes:
|
||||
- Configure the model.
|
||||
- Prepare data for inference.
|
||||
- Read resulting data after inference.
|
||||
This section describes how to handle dynamically shaped models with OpenVINO Runtime API version 2022.1 and higher. When using dynamic shapes, there are three main differences in the workflow than with static shapes:
|
||||
|
||||
* Configuring the model
|
||||
* Preparing and inferencing dynamic data
|
||||
* Dynamic shapes in outputs
|
||||
|
||||
### Configuring the Model
|
||||
|
||||
To avoid the methods mentioned in the previous section, there is a way to specify one or multiple dimensions to be dynamic, directly in the model inputs.
|
||||
This is achieved with the same reshape method that is used for alternating static shape of inputs.
|
||||
Dynamic dimensions are specified as `-1` or the `ov::Dimension()` instead of a positive number used for static dimensions:
|
||||
Model input dimensions can be specified as dynamic using the model.reshape method. To set a dynamic dimension, use `-1`, `ov::Dimension()` (C++), or `ov.Dimension()` (Python) as the value for that dimension.
|
||||
|
||||
> **NOTE**: Some models may already have dynamic shapes out of the box and do not require additional configuration. This can either be because it was generated with dynamic shapes from the source framework, or because it was converted with Model Optimizer to use dynamic shapes. For more information, see the Dynamic Dimensions “Out of the Box” section.
|
||||
|
||||
The examples below show how to set dynamic dimensions with a model that has a static `[1, 3, 224, 224]` input shape (such as [mobilenet-v2](https://docs.openvino.ai/2022.3/omz_models_model_mobilenet_v2.html)). The first example shows how to change the first dimension (batch size) to be dynamic. In the second example, the third and fourth dimensions (height and width) are set as dynamic.
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
|
@ -64,6 +66,8 @@ Dynamic dimensions are specified as `-1` or the `ov::Dimension()` instead of a p
|
|||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.py reshape_undefined
|
||||
|
||||
With Python, you may also pass all dimensions as a string and use `?` for the dynamic dimensions (e.g. `model.reshape(“1, 3, ?, ?”)`).
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{C}
|
||||
|
|
@ -74,45 +78,84 @@ Dynamic dimensions are specified as `-1` or the `ov::Dimension()` instead of a p
|
|||
|
||||
@endsphinxtabset
|
||||
|
||||
To simplify the code, the examples assume that the model has a single input and single output.
|
||||
However, there are no limitations on the number of inputs and outputs to apply dynamic shapes.
|
||||
|
||||
### Undefined Dimensions "Out Of the Box"
|
||||
|
||||
Dynamic dimensions may appear in the input model without calling the `reshape` method.
|
||||
Many DL frameworks support undefined dimensions.
|
||||
If such a model is converted with Model Optimizer or read directly by the `Core::read_model`, undefined dimensions are preserved.
|
||||
Such dimensions are automatically treated as dynamic ones.
|
||||
Therefore, there is no need to call the `reshape` method, if undefined dimensions are already configured in the original or the IR model.
|
||||
|
||||
If the input model has undefined dimensions that will not change during inference. It is recommended to set them to static values, using the same `reshape` method of the model.
|
||||
From the API perspective, any combination of dynamic and static dimensions can be configured.
|
||||
|
||||
Model Optimizer provides identical capability to reshape the model during the conversion, including specifying dynamic dimensions.
|
||||
Use this capability to save time on calling `reshape` method in the end application.
|
||||
To get information about setting input shapes using Model Optimizer, refer to [Setting Input Shapes](../MO_DG/prepare_model/convert_model/Converting_Model.md).
|
||||
|
||||
### Dimension Bounds
|
||||
|
||||
Apart from a dynamic dimension, the lower and/or upper bounds can also be specified. They define a range of allowed values for the dimension.
|
||||
The bounds are coded as arguments for the `ov::Dimension`:
|
||||
The examples above assume that the model has a single input layer. To change models with multiple input layers (such as NLP models), iterate over all the input layers, update the shape per layer, and apply the model.reshape method. For example, the following code sets the second dimension as dynamic in every input layer:
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@sphinxtab{C++}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_multiple_inputs
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{Python}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.py reshape_multiple_inputs
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
For more examples of how to change multiple input layers, see [Changing Input Shapes](ShapeInference.md).
|
||||
|
||||
#### Undefined Dimensions "Out Of the Box"
|
||||
|
||||
Many DL frameworks support generating models with dynamic (or undefined) dimensions. If such a model is converted with Model Optimizer or read directly by `Core::read_model`, its dynamic dimensions are preserved. These models do not need any additional configuration to use them with dynamic shapes.
|
||||
|
||||
To check if a model already has dynamic dimensions, first load it with the `read_model()` method, then check the `partial_shape` property of each layer. If the model has any dynamic dimensions, they will be reported as `?`. For example, the following code will print the name and dimensions of each input layer:
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@sphinxtab{C++}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:check_inputs
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{Python}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.py check_inputs
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
If the input model already has dynamic dimensions, that will not change during inference. If the inputs will not be used dynamically, it is recommended to set them to static values using the `reshape` method to save application memory and potentially improve inference speed. The OpenVINO API supports any combination of static and dynamic dimensions.
|
||||
|
||||
Static and dynamic dimensions can also be set when converting the model with Model Optimizer. It has identical capabilities to the ``reshape`` method, so you can save time by converting the model with dynamic shapes beforehand rather than in the application code. To get information about setting input shapes using Model Optimizer, refer to [Setting Input Shapes](../MO_DG/prepare_model/convert_model/Converting_Model.md).
|
||||
|
||||
#### Dimension Bounds
|
||||
|
||||
The lower and/or upper bounds of a dynamic dimension can also be specified. They define a range of allowed values for the dimension. Dimension bounds can be set by passing the lower and upper bounds into the `reshape` method using the options shown below.
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@sphinxtab{C++}
|
||||
|
||||
The dimension bounds can be coded as arguments for `ov::Dimension`, as shown in these examples:
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_bounds
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{Python}
|
||||
|
||||
Each of these options are equivalent:
|
||||
|
||||
- Pass the lower and upper bounds directly into the `reshape` method, e.g. `model.reshape([1, 10), (8,512)])`
|
||||
- Pass the lower and upper bounds using ov.Dimension, e.g. `model.reshape([ov.Dimension(1, 10), (8, 512)])`
|
||||
- Pass the dimension ranges as strings, e.g. `model.reshape(“1..10, 8..512”)`
|
||||
|
||||
The examples below show how to set dynamic dimension bounds for a mobilenet-v2 model with a default static shape of `[1,3,224,224]`.
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.py reshape_bounds
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{C}
|
||||
|
||||
The dimension bounds can be coded as arguments for [ov_dimension](https://docs.openvino.ai/2022.3/structov_dimension.html#doxid-structov-dimension), as shown in these examples:
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.c ov_dynamic_shapes:reshape_bounds
|
||||
|
||||
@endsphinxtab
|
||||
|
|
@ -131,11 +174,11 @@ Depending on the plugin, specifying the upper bounds can be required. For inform
|
|||
|
||||
If the lower and upper bounds for a dimension are known, it is recommended to specify them, even if a plugin can execute a model without the bounds.
|
||||
|
||||
### Setting Input Tensors
|
||||
### Preparing and Inferencing Dynamic Data
|
||||
|
||||
Preparing a model with the `reshape` method is the first step.
|
||||
The second step is passing a tensor with an appropriate shape to infer request.
|
||||
This is similar to the [regular steps](integrate_with_your_application.md). However, tensors can now be passed with different shapes for the same executable model and even for the same inference request:
|
||||
After configuring a model with the `reshape` method, the next steps are to create tensors with the appropriate data shape and pass them to the model as an inference request. This is similar to the regular steps described in [Integrate OpenVINO™ with Your Application](integrate_with_your_application.md). However, tensors can now be passed into the model with different shapes.
|
||||
|
||||
The sample below shows how a model can accept different input shapes. In the first case, the model runs inference on a 1x128 input shape and returns a result. In the second case, a 1x200 input shape is used, which the model can still handle because it is dynamically shaped.
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
|
@ -159,45 +202,13 @@ This is similar to the [regular steps](integrate_with_your_application.md). Howe
|
|||
|
||||
@endsphinxtabset
|
||||
|
||||
In the example above, the `set_input_tensor` is used to specify input tensors.
|
||||
The real dimension of the tensor is always static, because it is a particular tensor and it does not have any dimension variations in contrast to model inputs.
|
||||
|
||||
Similar to static shapes, `get_input_tensor` can be used instead of `set_input_tensor`.
|
||||
In contrast to static input shapes, when using `get_input_tensor` for dynamic inputs, the `set_shape` method for the returned tensor should be called to define the shape and allocate memory.
|
||||
Without doing so, the tensor returned by `get_input_tensor` is an empty tensor. The shape of the tensor is not initialized and memory is not allocated, because infer request does not have information about the real shape that will be provided.
|
||||
Setting shape for an input tensor is required when the corresponding input has at least one dynamic dimension, regardless of the bounds.
|
||||
Contrary to previous example, the following one shows the same sequence of two infer requests, using `get_input_tensor` instead of `set_input_tensor`:
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@sphinxtab{C++}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:get_input_tensor
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{Python}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.py get_input_tensor
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@sphinxtab{C}
|
||||
|
||||
@snippet docs/snippets/ov_dynamic_shapes.c ov_dynamic_shapes:get_input_tensor
|
||||
|
||||
@endsphinxtab
|
||||
|
||||
@endsphinxtabset
|
||||
For more information on how to apply input data to a model and run inference, see [OpenVINO™ Inference Request](ov_infer_request.md).
|
||||
|
||||
### Dynamic Shapes in Outputs
|
||||
|
||||
Examples above are valid approaches when dynamic dimensions in output may be implied by propagation of dynamic dimension from the inputs.
|
||||
For example, batch dimension in an input shape is usually propagated through the whole model and appears in the output shape.
|
||||
It also applies to other dimensions, like sequence length for NLP models or spatial dimensions for segmentation models, that are propagated through the entire network.
|
||||
When using dynamic dimensions in the input of a model, one or more output dimensions may also be dynamic depending on how the dynamic inputs are propagated through the model. For example, the batch dimension in an input shape is usually propagated through the whole model and appears in the output shape. It also applies to other dimensions, like sequence length for NLP models or spatial dimensions for segmentation models, that are propagated through the entire network.
|
||||
|
||||
Whether the output has dynamic dimensions or not can be verified by querying the output partial shape after the model is read or reshaped.
|
||||
The same applies to inputs. For example:
|
||||
To determine if the output has dynamic dimensions, the `partial_shape` property of the model’s output layers can be queried after the model has been read or reshaped. The same property can be queried for model inputs. For example:
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
|
@ -221,9 +232,9 @@ The same applies to inputs. For example:
|
|||
|
||||
@endsphinxtabset
|
||||
|
||||
When there are dynamic dimensions in corresponding inputs or outputs, the `?` or ranges like `1..10` appear.
|
||||
If the output has any dynamic dimensions, they will be reported as `?` or as a range (e.g.`1..10`).
|
||||
|
||||
It can also be verified in a more programmatic way:
|
||||
Output layers can also be checked for dynamic dimensions using the `partial_shape.is_dynamic()` property. This can be used on an entire output layer, or on an individual dimension, as shown in these examples:
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
|
@ -247,8 +258,6 @@ It can also be verified in a more programmatic way:
|
|||
|
||||
@endsphinxtabset
|
||||
|
||||
If at least one dynamic dimension exists in the output layer of a model, the actual shape of the output tensor will be determined during inference. Before the first inference, the output tensor’s memory is not allocated and has a shape of `[0]`.
|
||||
|
||||
If at least one dynamic dimension exists in an output of a model, a shape of the corresponding output tensor will be set as the result of inference call.
|
||||
Before the first inference, memory for such a tensor is not allocated and has the `[0]` shape.
|
||||
If the `set_output_tensor` method is called with a pre-allocated tensor, the inference will call the `set_shape` internally, and the initial shape is replaced by the calculated shape.
|
||||
Therefore, setting a shape for output tensors in this case is useful only when pre-allocating enough memory for output tensor. Normally, the `set_shape` method of a `Tensor` re-allocates memory only if a new shape requires more storage.
|
||||
To pre-allocate space in memory for the output tensor, use the `set_output_tensor` method with the expected shape of the output. This will call the `set_shape` method internally, which will cause the initial shape to be replaced by the calculated shape.
|
||||
|
|
|
|||
|
|
@ -9,15 +9,14 @@ Previously, a certain level of automatic configuration was the result of the *de
|
|||
The hints, in contrast, respect the actual model, so the parameters for optimal throughput are calculated for each model individually (based on its compute versus memory bandwidth requirements and capabilities of the device).
|
||||
|
||||
## Performance Hints: Latency and Throughput
|
||||
As discussed in the [Optimization Guide](../optimization_guide/dldt_deployment_optimization_guide.md) there are a few different metrics associated with inference speed.
|
||||
Throughput and latency are some of the most widely used metrics that measure the overall performance of an application.
|
||||
|
||||
Therefore, in order to ease the configuration of the device, OpenVINO offers two dedicated hints, namely `ov::hint::PerformanceMode::THROUGHPUT` and `ov::hint::PerformanceMode::LATENCY`.
|
||||
A special `ov::hint::PerformanceMode::UNDEFINED` hint acts the same as specifying no hint.
|
||||
As discussed in the [Optimization Guide](../optimization_guide/dldt_deployment_optimization_guide.md) there are a few different metrics associated with inference speed. Throughput and latency are some of the most widely used metrics that measure the overall performance of an application.
|
||||
|
||||
Therefore, in order to ease the configuration of the device, OpenVINO offers two dedicated hints, namely `ov::hint::PerformanceMode::THROUGHPUT` and `ov::hint::PerformanceMode::LATENCY`. A special `ov::hint::PerformanceMode::UNDEFINED` hint acts the same as specifying no hint.
|
||||
|
||||
For more information on conducting performance measurements with the `benchmark_app`, refer to the last section in this document.
|
||||
|
||||
Keep in mind that a typical model may take significantly more time to load with the `ov::hint::PerformanceMode::THROUGHPUT` and consume much more memory, compared to the `ov::hint::PerformanceMode::LATENCY`.
|
||||
Keep in mind that a typical model may take significantly more time to load with the `ov::hint::PerformanceMode::THROUGHPUT` and consume much more memory, compared to the `ov::hint::PerformanceMode::LATENCY`. Also, the `THROUGHPUT` and `LATENCY` hints only improve performance in an asynchronous inference pipeline. For information on asynchronous inference, see the [Prefer Async API](#prefer-async-api) section of this document.
|
||||
|
||||
## Performance Hints: How It Works
|
||||
Internally, every device "translates" the value of the hint to the actual performance settings.
|
||||
|
|
@ -99,9 +98,13 @@ While an application is free to create more requests if needed (for example to s
|
|||
Keep in mind that `ov::hint::PerformanceMode::LATENCY` does not necessarily imply using single inference request. For example, multi-socket CPUs can deliver as many requests at the same minimal latency as the number of NUMA nodes in the system.
|
||||
To make your application fully scalable, make sure to query the `ov::optimal_number_of_infer_requests` directly.
|
||||
|
||||
## Prefer Async API
|
||||
The API of the inference requests offers Sync and Async execution. The `ov::InferRequest::infer()` is inherently synchronous and simple to operate (as it serializes the execution flow in the current application thread). The Async "splits" the `infer()` into `ov::InferRequest::start_async()` and `ov::InferRequest::wait()` (or callbacks). For more information, refer to the [API examples](../OV_Runtime_UG/ov_infer_request.md).
|
||||
Although the Synchronous API can be somewhat easier to start with, it is recommended to use the Asynchronous (callbacks-based) API in the production code. It is the most general and scalable way to implement the flow control for any possible number of requests (and thus both latency and throughput scenarios).
|
||||
## <a name="prefer-async-api"></a>Prefer Async API
|
||||
|
||||
The API of the inference requests offers Sync and Async execution. The `ov::InferRequest::infer()` is inherently synchronous and simple to operate (as it serializes the execution flow in the current application thread). The Async "splits" the `infer()` into `ov::InferRequest::start_async()` and `ov::InferRequest::wait()` (or callbacks). For more information on synchronous and asynchronous modes, refer to the [OpenVINO Inference Request documentation](../OV_Runtime_UG/ov_infer_request.md).
|
||||
|
||||
Although the synchronous API can be easier to start with, it is recommended to use the asynchronous (callbacks-based) API in production code. It is the most general and scalable way to implement the flow control for any possible number of requests. The `THROUGHPUT` and `LATENCY` performance hints automatically configure the Asynchronous pipeline to use the optimal number of processing streams and inference requests.
|
||||
|
||||
> **NOTE**: **Important:** Performance Hints only work when asynchronous execution mode is used. They do not affect the performance of a synchronous pipeline.
|
||||
|
||||
## Combining the Hints and Individual Low-Level Settings
|
||||
While sacrificing the portability to some extent, it is possible to combine the hints with individual device-specific settings.
|
||||
|
|
@ -127,8 +130,8 @@ For example, use `ov::hint::PerformanceMode::THROUGHPUT` to prepare a general co
|
|||
The `benchmark_app`, that exists in both [C++](../../samples/cpp/benchmark_app/README.md) and [Python](../../tools/benchmark_tool/README.md) versions, is the best way to evaluate the functionality of the performance hints for a particular device:
|
||||
- benchmark_app **-hint tput** -d 'device' -m 'path to your model'
|
||||
- benchmark_app **-hint latency** -d 'device' -m 'path to your model'
|
||||
- Disabling the hints to emulate the pre-hints era (highly recommended before trying the individual low-level settings, such as the number of streams as below, threads, etc):
|
||||
- - benchmark_app **-hint none -nstreams 1** -d 'device' -m 'path to your model'
|
||||
- Disabling the hints to emulate the pre-hints era (highly recommended before trying the individual low-level settings, such as the number of streams as below, threads, etc):
|
||||
- benchmark_app **-hint none -nstreams 1** -d 'device' -m 'path to your model'
|
||||
|
||||
|
||||
### Additional Resources
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
# CPU Device {#openvino_docs_OV_UG_supported_plugins_CPU}
|
||||
|
||||
The CPU plugin is a part of the Intel® Distribution of OpenVINO™ toolkit. It is developed to achieve high performance inference of neural networks on Intel® x86-64 CPUs.
|
||||
For an in-depth description of CPU plugin, see:
|
||||
The CPU plugin is a part of the Intel® Distribution of OpenVINO™ toolkit. It is developed to achieve high performance inference of neural networks on Intel® x86-64 CPUs.The newer 11th generation and later Intel® CPUs provide even further performance boost, especially with INT8 models.
|
||||
|
||||
- [CPU plugin developers documentation](https://github.com/openvinotoolkit/openvino/wiki/CPUPluginDevelopersDocs).
|
||||
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@ For more details on how to get a quantized model, refer to the [Model Optimizati
|
|||
|
||||
Floating-point precision of a GPU primitive is selected based on operation precision in the OpenVINO IR, except for the [compressed f16 OpenVINO IR form](../../MO_DG/prepare_model/FP16_Compression.md), which is executed in the `f16` precision.
|
||||
|
||||
> **NOTE**: Hardware acceleration for `i8`/`u8` precision may be unavailable on some platforms. In such cases, a model is executed in the floating-point precision taken from IR. Hardware support of `u8`/`i8` acceleration can be queried via the `ov::device::capabilities` property.
|
||||
> **NOTE**: The newer generation Intel Iris Xe and Xe MAX GPUs provide accelerated performance for i8/u8 models. Hardware acceleration for i8/u8 precision may be unavailable on older generation platforms. In such cases, a model is executed in the floating-point precision taken from IR. Hardware support of u8/i8 acceleration can be queried via the `ov::device::capabilities` property.
|
||||
|
||||
[Hello Query Device C++ Sample](../../../samples/cpp/hello_query_device/README.md) can be used to print out the supported data types for all detected devices.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
Supported Devices {#openvino_docs_OV_UG_supported_plugins_Supported_Devices}
|
||||
==================
|
||||
|
||||
The OpenVINO Runtime can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device. In OpenVINO™ documentation, "device" refers to an Intel® processors used for inference, which can be a supported CPU, GPU, VPU (vision processing unit), or GNA (Gaussian neural accelerator coprocessor), or a combination of those devices.
|
||||
The OpenVINO runtime can infer various models of different input and output formats. Here, you can find configurations
|
||||
supported by OpenVINO devices, which are CPU, GPU, or GNA (Gaussian neural accelerator coprocessor). Currently, 11th generation and later processors (currently up to 13th generation) provide a further performance boost, especially with INT8 models.
|
||||
|
||||
> **NOTE**: With OpenVINO™ 2020.4 release, Intel® Movidius™ Neural Compute Stick support has been cancelled.
|
||||
|
||||
|
|
|
|||
|
|
@ -12,21 +12,21 @@ ov_core_create(&core);
|
|||
ov_model_t* model = NULL;
|
||||
ov_core_read_model(core, "model.xml", NULL, &model);
|
||||
|
||||
// Set one static dimension (= 1) and another dynamic dimension (= Dimension())
|
||||
// Set first dimension as dynamic ({-1, -1}) and remaining dimensions as static
|
||||
{
|
||||
ov_partial_shape_t partial_shape;
|
||||
ov_dimension_t dims[2] = {{1, 1}, {-1, -1}};
|
||||
ov_partial_shape_create(2, dims, &partial_shape);
|
||||
ov_model_reshape_single_input(model, partial_shape); // {1,?}
|
||||
ov_dimension_t dims[4] = {{-1, -1}, {3, 3}, {224, 224}, {224, 224}};
|
||||
ov_partial_shape_create(4, dims, &partial_shape);
|
||||
ov_model_reshape_single_input(model, partial_shape); // {?,3,224,224}
|
||||
ov_partial_shape_free(&partial_shape);
|
||||
}
|
||||
|
||||
// Or set both dimensions as dynamic if both are going to be changed dynamically
|
||||
// Or, set third and fourth dimensions as dynamic
|
||||
{
|
||||
ov_partial_shape_t partial_shape;
|
||||
ov_dimension_t dims[2] = {{-1, -1}, {-1, -1}};
|
||||
ov_partial_shape_create(2, dims, &partial_shape);
|
||||
ov_model_reshape_single_input(model, partial_shape); // {?,?}
|
||||
ov_dimension_t dims[4] = {{1, 1}, {3, 3}, {-1, -1}, {-1, -1}};
|
||||
ov_partial_shape_create(4, dims, &partial_shape);
|
||||
ov_model_reshape_single_input(model, partial_shape); // {1,3,?,?}
|
||||
ov_partial_shape_free(&partial_shape);
|
||||
}
|
||||
//! [ov_dynamic_shapes:reshape_undefined]
|
||||
|
|
|
|||
|
|
@ -10,17 +10,11 @@ void reshape_with_dynamics() {
|
|||
ov::Core core;
|
||||
auto model = core.read_model("model.xml");
|
||||
|
||||
// Set one static dimension (= 1) and another dynamic dimension (= Dimension())
|
||||
model->reshape({{1, ov::Dimension()}}); // {1,?}
|
||||
// Set first dimension as dynamic (ov::Dimension()) and remaining dimensions as static
|
||||
model->reshape({{ov::Dimension(), 3, 224, 224}}); // {?,3,224,224}
|
||||
|
||||
// The same as above
|
||||
model->reshape({{1, -1}}); // {1,?}
|
||||
|
||||
// Or set both dimensions as dynamic if both are going to be changed dynamically
|
||||
model->reshape({{ov::Dimension(), ov::Dimension()}}); // {?,?}
|
||||
|
||||
// The same as above
|
||||
model->reshape({{-1, -1}}); // {?,?}
|
||||
// Or, set third and fourth dimensions as dynamic
|
||||
model->reshape({{1, 3, ov::Dimension(), ov::Dimension()}}); // {1,3,?,?}
|
||||
//! [ov_dynamic_shapes:reshape_undefined]
|
||||
//! [ov_dynamic_shapes:reshape_bounds]
|
||||
// Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
|
||||
|
|
@ -59,6 +53,34 @@ if (model->output(0).get_partial_shape()[1].is_dynamic()) {
|
|||
}
|
||||
//! [ov_dynamic_shapes:detect_dynamic]
|
||||
}
|
||||
{
|
||||
//! [ov_dynamic_shapes:check_inputs]
|
||||
ov::Core core;
|
||||
auto model = core.read_model("model.xml");
|
||||
|
||||
// Print info of first input layer
|
||||
std::cout << model->input(0).get_partial_shape() << "\n";
|
||||
|
||||
// Print info of second input layer
|
||||
std::cout << model->input(1).get_partial_shape() << "\n";
|
||||
|
||||
//etc
|
||||
//! [ov_dynamic_shapes:check_inputs]
|
||||
}
|
||||
{
|
||||
ov::Core core;
|
||||
auto model = core.read_model("model.xml");
|
||||
//! [ov_dynamic_shapes:reshape_multiple_inputs]
|
||||
// Assign dynamic shapes to second dimension in every input layer
|
||||
std::map<ov::Output<ov::Node>, ov::PartialShape> port_to_shape;
|
||||
for (const ov::Output<ov::Node>& input : model->inputs()) {
|
||||
ov::PartialShape shape = input.get_partial_shape();
|
||||
shape[1] = -1;
|
||||
port_to_shape[input] = shape;
|
||||
}
|
||||
model->reshape(port_to_shape);
|
||||
//! [ov_dynamic_shapes:reshape_multiple_inputs]
|
||||
}
|
||||
}
|
||||
|
||||
void set_tensor() {
|
||||
|
|
|
|||
|
|
@ -7,40 +7,22 @@ import openvino.runtime as ov
|
|||
#! [import]
|
||||
|
||||
#! [reshape_undefined]
|
||||
core = ov.Core()
|
||||
model = core.read_model("model.xml")
|
||||
Core = ov.Core()
|
||||
model = core.read_model(“model.xml”)
|
||||
|
||||
# Set one static dimension (= 1) and another dynamic dimension (= Dimension())
|
||||
model.reshape([1, ov.Dimension()])
|
||||
# Set first dimension to be dynamic while keeping others static
|
||||
model.reshape([-1, 3, 224, 224])
|
||||
|
||||
# The same as above
|
||||
model.reshape([1, -1])
|
||||
|
||||
# The same as above
|
||||
model.reshape("1, ?")
|
||||
|
||||
# Or set both dimensions as dynamic if both are going to be changed dynamically
|
||||
model.reshape([ov.Dimension(), ov.Dimension()])
|
||||
|
||||
# The same as above
|
||||
model.reshape([-1, -1])
|
||||
|
||||
# The same as above
|
||||
model.reshape("?, ?")
|
||||
# Or, set third and fourth dimensions as dynamic
|
||||
model.reshape([1, 3, -1, -1])
|
||||
#! [reshape_undefined]
|
||||
|
||||
#! [reshape_bounds]
|
||||
# Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
|
||||
model.reshape([ov.Dimension(1, 10), ov.Dimension(8, 512)])
|
||||
# Example 1 - set first dimension as dynamic (no bounds) and third and fourth dimensions to range of 112..448
|
||||
model.reshape([-1, 3, (112, 448), (112, 448)])
|
||||
|
||||
# The same as above
|
||||
model.reshape([(1, 10), (8, 512)])
|
||||
|
||||
# The same as above
|
||||
model.reshape("1..10, 8..512")
|
||||
|
||||
# Both dimensions are dynamic, first doesn't have bounds, the second is in the range of 8..512
|
||||
model.reshape([-1, (8, 512)])
|
||||
# Example 2 - Set first dimension to a range of 1..8 and third and fourth dimensions to range of 112..448
|
||||
model.reshape([(1, 8), 3, (112, 448), (112, 448)])
|
||||
#! [reshape_bounds]
|
||||
|
||||
model = core.read_model("model.xml")
|
||||
|
|
@ -73,45 +55,21 @@ executable = core.compile_model(model)
|
|||
infer_request = executable.create_infer_request()
|
||||
|
||||
#! [set_input_tensor]
|
||||
# The first inference call
|
||||
# For first inference call, prepare an input tensor with 1x128 shape and run inference request
|
||||
Input_data1 = np.ones(shape=[1,128])
|
||||
infer_request.infer([input_data1])
|
||||
|
||||
# Create tensor compatible to the model input
|
||||
# Shape {1, 128} is compatible with any reshape statements made in previous examples
|
||||
input_tensor1 = ov.Tensor(model.input().element_type, [1, 128])
|
||||
# ... write values to input_tensor_1
|
||||
# Get resulting outputs
|
||||
Output_tensor1 = infer_request.get_output_tensor()
|
||||
Output_data1 = output_tensor.data[:]
|
||||
|
||||
# Set the tensor as an input for the infer request
|
||||
infer_request.set_input_tensor(input_tensor1)
|
||||
# For second inference call, prepare a 1x200 input tensor and run inference request
|
||||
Input_data2 = np.ones(shape=[1,200])
|
||||
infer_request.infer([input_data2])
|
||||
|
||||
# Do the inference
|
||||
infer_request.infer()
|
||||
|
||||
# Or pass a tensor in infer to set the tensor as a model input and make the inference
|
||||
infer_request.infer([input_tensor1])
|
||||
|
||||
# Or pass the numpy array to set inputs of the infer request
|
||||
input_data = np.ones(shape=[1, 128])
|
||||
infer_request.infer([input_data])
|
||||
|
||||
# Retrieve a tensor representing the output data
|
||||
output_tensor = infer_request.get_output_tensor()
|
||||
|
||||
# Copy data from tensor to numpy array
|
||||
data1 = output_tensor.data[:]
|
||||
|
||||
# The second inference call, repeat steps:
|
||||
|
||||
# Create another tensor (if the previous one cannot be utilized)
|
||||
# Notice, the shape is different from input_tensor_1
|
||||
input_tensor2 = ov.Tensor(model.input().element_type, [1, 200])
|
||||
# ... write values to input_tensor_2
|
||||
|
||||
infer_request.infer([input_tensor2])
|
||||
|
||||
# No need to call infer_request.get_output_tensor() again
|
||||
# output_tensor queried after the first inference call above is valid here.
|
||||
# But it may not be true for the memory underneath as shape changed, so re-take an output data:
|
||||
data2 = output_tensor.data[:]
|
||||
# Get resulting outputs
|
||||
Output_tensor2 = infer_request.get_output_tensor()
|
||||
Output_data2 = output_tensor.data[:]
|
||||
#! [set_input_tensor]
|
||||
|
||||
infer_request = executable.create_infer_request()
|
||||
|
|
@ -137,3 +95,21 @@ input_tensor.shape = [1, 200]
|
|||
infer_request.infer()
|
||||
data2 = output_tensor.data[:]
|
||||
#! [get_input_tensor]
|
||||
|
||||
#! [check_inputs]
|
||||
core = ov.Core()
|
||||
model = core.read_model("model.xml")
|
||||
|
||||
# Print model input layer info
|
||||
for input_layer in model.inputs:
|
||||
print(input_layer.names, input_layer.partial_shape)
|
||||
#! [check_inputs]
|
||||
|
||||
#! [reshape_multiple_inputs]
|
||||
# Assign dynamic shapes to second dimension in every input layer
|
||||
shapes = {}
|
||||
for input_layer in model.inputs:
|
||||
shapes[input_layer] = input_layer.partial_shape
|
||||
shapes[input_layer][1] = -1
|
||||
model.reshape(shapes)
|
||||
#! [reshape_multiple_inputs]
|
||||
|
|
|
|||
Loading…
Reference in New Issue