natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] ref fix + benchmark fix + ov:state_network fix (#22529)

This commit is contained in:
Maciej Smyk 2024-01-30 10:38:47 +01:00 committed by GitHub
parent ec4f36b015
commit ffb5270986
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
11 changed files with 205 additions and 205 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/articles_en/about_openvino/performance_benchmarks.rst
View File

@ -133,7 +133,7 @@ For a listing of all platforms and configurations used for testing, refer to the
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. button-link:: _static/benchmarks_files/benchmarks_files/OV-2023.3-Performance-Data.xlsx
.. button-link:: _static/benchmarks_files/OV-2023.3-Performance-Data.xlsx
:color: primary
:outline:
:expand:

36
docs/articles_en/documentation/openvino_extensibility.rst
View File

@ -5,8 +5,8 @@ OpenVINO Extensibility Mechanism
.. meta::
:description: Explore OpenVINO™ Extensibility API, which allows adding
support for models with custom operations and their further implementation
:description: Explore OpenVINO™ Extensibility API, which allows adding
support for models with custom operations and their further implementation
in applications.
.. toctree::
@ -16,18 +16,18 @@ OpenVINO Extensibility Mechanism
openvino_docs_Extensibility_UG_add_openvino_ops
openvino_docs_Extensibility_UG_Frontend_Extensions
openvino_docs_Extensibility_UG_GPU
.. toctree::
:maxdepth: 1
:hidden:
openvino_docs_transformations
OpenVINO Plugin Developer Guide <openvino_docs_ie_plugin_dg_overview>
The Intel® Distribution of OpenVINO™ toolkit supports neural-network models trained with various frameworks, including
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle (OpenVINO support for Apache MXNet, Caffe, and Kaldi is currently
being deprecated and will be removed entirely in the future). The list of supported operations is different for each of the supported frameworks.
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle (OpenVINO support for Apache MXNet, Caffe, and Kaldi is currently
being deprecated and will be removed entirely in the future). The list of supported operations is different for each of the supported frameworks.
To see the operations supported by your framework, refer to :doc:`Supported Framework Operations <openvino_resources_supported_operations_frontend>`.
Custom operations, which are not included in the list, are not recognized by OpenVINO out-of-the-box. The need for custom operation may appear in two cases:
@ -80,32 +80,32 @@ Registering Extensions
A custom operation class and a new mapping frontend extension class object should be registered to be usable in OpenVINO runtime.
.. note::
.. note::
This documentation is derived from the `Template extension <https://github.com/openvinotoolkit/openvino/tree/master/src/core/template_extension/new>`__, which demonstrates the details of extension development. It is based on minimalistic ``Identity`` operation that is a placeholder for your real custom operation. Review the complete, fully compilable code to see how it works.
Use the ``:ref:`ov::Core::add_extension <doxid-classov_1_1_core_1a68d0dea1cbcd42a67bea32780e32acea>``` method to load the extensions to the ``:ref:`ov::Core <doxid-classov_1_1_core>``` object. This method allows loading library with extensions or extensions from the code.
Use the ``ov::Core::add_extension`` method to load the extensions to the ``ov::Core`` object. This method allows loading library with extensions or extensions from the code.
Load Extensions to Core
+++++++++++++++++++++++
Extensions can be loaded from a code with the ``:ref:`ov::Core::add_extension <doxid-classov_1_1_core_1a68d0dea1cbcd42a67bea32780e32acea>``` method:
Extensions can be loaded from a code with the ``ov::Core::add_extension`` method:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_extensions.py
:language: python
:fragment: [add_extension]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_extensions.cpp
:language: cpp
:fragment: [add_extension]
The ``Identity`` is a custom operation class defined in :doc:`Custom Operation Guide <openvino_docs_Extensibility_UG_add_openvino_ops>`. This is sufficient to enable reading OpenVINO IR which uses the ``Identity`` extension operation emitted by Model Optimizer. In order to load original model directly to the runtime, add a mapping extension:
@ -130,7 +130,7 @@ When Python API is used, there is no way to implement a custom OpenVINO operatio
Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used.
.. _create_a_library_with_extensions:
Create a Library with Extensions
++++++++++++++++++++++++++++++++
@ -142,7 +142,7 @@ An extension library should be created in the following cases:
To create an extension library, for example, to load the extensions into Model Optimizer, perform the following:
1. Create an entry point for extension library. OpenVINO provides the ``:ref:`OPENVINO_CREATE_EXTENSIONS() <doxid-core_2include_2openvino_2core_2extension_8hpp_1acdadcfa0eff763d8b4dadb8a9cb6f6e6>``` macro, which allows to define an entry point to a library with OpenVINO Extensions.
1. Create an entry point for extension library. OpenVINO provides the ``OPENVINO_CREATE_EXTENSIONS()`` macro, which allows to define an entry point to a library with OpenVINO Extensions.
This macro should have a vector of all OpenVINO Extensions as an argument.
Based on that, the declaration of an extension class might look like the following:
@ -162,7 +162,7 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command.
3. Build the extension library, running the commands below:
.. code-block:: sh
$ cd src/core/template_extension/new
$ mkdir build
$ cd build
@ -173,16 +173,16 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command.
4. After the build, you may use the path to your extension library to load your extensions to OpenVINO Runtime:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_extensions.py
:language: python
:fragment: [add_extension_lib]
.. tab-item:: C++
:sync: cpp
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_extensions.cpp
:language: cpp

14
docs/articles_en/documentation/openvino_extensibility/GPU_Extensibility.rst
View File

@ -5,7 +5,7 @@ How to Implement Custom GPU Operations
.. meta::
:description: Learn the details of custom kernel support for the GPU device to
:description: Learn the details of custom kernel support for the GPU device to
enable operations not supported by OpenVINO.
@ -16,20 +16,20 @@ The GPU codepath abstracts many details about OpenCL. You need to provide the ke
There are two options for using the custom operation configuration file:
* Include a section with your kernels into the automatically-loaded ``<lib_path>/cldnn_global_custom_kernels/cldnn_global_custom_kernels.xml`` file.
* Call the ``:ref:`ov::Core::set_property() <doxid-classov_1_1_core_1aa953cb0a1601dbc9a34ef6ba82b8476e>``` method from your application with the ``"CONFIG_FILE"`` key and the configuration file name as a value before loading the network that uses custom operations to the plugin:
* Call the ``ov::Core::set_property()`` method from your application with the ``"CONFIG_FILE"`` key and the configuration file name as a value before loading the network that uses custom operations to the plugin:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.py
:language: python
:fragment: [part0]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.cpp
:language: cpp
:fragment: [part0]
@ -43,10 +43,10 @@ feature a dedicated command-line option ``-c`` to load custom kernels. For examp
$ ./classification_sample -m <path_to_model>/bvlc_alexnet_fp16.xml -i ./validation_set/daily/227x227/apron.bmp -d GPU
-c <absolute_path_to_config>/custom_layer_example.xml
.. _config-file-format:
.. _config-file-format:
Configuration File Format
#########################
#########################
The configuration file is expected to follow the ``.xml`` file structure
with a node of the type ``CustomLayer`` for every custom operation you provide.
@ -344,7 +344,7 @@ Example Kernel
.. _debugging-tips:
.. note::
.. note::
As described in the previous section, all items such as the ``INPUT0_TYPE`` are actually defined as OpenCL (pre-)compiler inputs by OpenVINO for efficiency reasons. See the `Debugging Tips <#debugging-tips>`__ below for information on debugging the results.
Debugging Tips

22
docs/articles_en/documentation/openvino_extensibility/add_openvino_ops.rst
View File

@ -5,8 +5,8 @@ Custom OpenVINO™ Operations
.. meta::
:description: Explore OpenVINO™ Extension API which enables registering
custom operations to support models with operations
:description: Explore OpenVINO™ Extension API which enables registering
custom operations to support models with operations
not supported by OpenVINO.
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_library_with_extensions>` for more details on library creation and usage. The remining part of this document describes how to implement an operation class.
@ -14,7 +14,7 @@ OpenVINO™ Extension API allows you to register custom operations to support mo
Operation Class
###############
To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``:ref:`ov::Node <doxid-classov_1_1_node>```, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
.. doxygensnippet:: ./src/core/template_extension/new/identity.hpp
:language: cpp
@ -24,9 +24,9 @@ Follow the steps below to add a custom operation:
1. Add the ``OPENVINO_OP`` macro which defines a ``NodeTypeInfo`` object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an operation currently consists of a string operation identifier and a string for operation version.
2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``:ref:`ov::Node <doxid-classov_1_1_node>```. Set the inferred shape and element type of the output using ``set_output_type``.
3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.
4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.
@ -40,9 +40,9 @@ Based on that, declaration of an operation class can look as follows:
Operation Constructors
++++++++++++++++++++++
OpenVINO™ operation contains two constructors:
OpenVINO™ operation contains two constructors:
* Default constructor, which enables you to create an operation without attributes
* Default constructor, which enables you to create an operation without attributes
* Constructor that creates and validates an operation with specified inputs and attributes
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
@ -52,7 +52,7 @@ OpenVINO™ operation contains two constructors:
``validate_and_infer_types()``
++++++++++++++++++++++++++++++
``:ref:`ov::Node::validate_and_infer_types <doxid-classov_1_1_node_1ac5224b5be848ec670d2078d9816d12e7>``` method validates operation attributes and calculates output shapes using attributes of the operation.
``ov::Node::validate_and_infer_types`` method validates operation attributes and calculates output shapes using attributes of the operation.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -61,7 +61,7 @@ OpenVINO™ operation contains two constructors:
``clone_with_new_inputs()``
+++++++++++++++++++++++++++
``:ref:`ov::Node::clone_with_new_inputs <doxid-classov_1_1_node_1a04cb103fa069c3b7944ab7c44d94f5ff>``` method creates a copy of the operation with new inputs.
``ov::Node::clone_with_new_inputs`` method creates a copy of the operation with new inputs.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -70,7 +70,7 @@ OpenVINO™ operation contains two constructors:
``visit_attributes()``
++++++++++++++++++++++
``:ref:`ov::Node::visit_attributes <doxid-classov_1_1_node_1a9743b56d352970486d17dae2416d958e>``` method enables you to visit all operation attributes.
``ov::Node::visit_attributes`` method enables you to visit all operation attributes.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -79,7 +79,7 @@ OpenVINO™ operation contains two constructors:
``evaluate() and has_evaluate()``
+++++++++++++++++++++++++++++++++
``:ref:`ov::Node::evaluate <doxid-classov_1_1_node_1acfb82acc8349d7138aeaa05217c7014e>``` method enables you to apply constant folding to an operation.
``ov::Node::evaluate`` method enables you to apply constant folding to an operation.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp

50
docs/articles_en/documentation/openvino_extensibility/openvino_plugin_library/detailed_guides/low_precision_transformations/step2_markup.rst
View File

@ -5,7 +5,7 @@ Step 2. Markup Transformations
.. meta::
:description: Learn about markup transformations, which are used to create
:description: Learn about markup transformations, which are used to create
attributes for input and output ports and operations during runtime.
.. toctree::
@ -21,7 +21,7 @@ Step 2. Markup Transformations
PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>
AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>
AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>
CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>
CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>
PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>
@ -31,13 +31,13 @@ Step 2. Markup Transformations
This step defines the optimal ``FakeQuantize`` decomposition precisions for the best inference performance via operations markup with runtime attribute instances. Attributes are created for input and output ports and operations. Transformations do not change the operation output port precisions. A model markup low precision logic is decomposed and implemented into the following common markup transformations. The order of transformations is important:
1. :doc:`MarkupBias <openvino_docs_OV_UG_lpt_MarkupBias>`
2. :doc:`MarkupCanBeQuantized <openvino_docs_OV_UG_lpt_MarkupCanBeQuantized>`
3. :doc:`MarkupPrecisions <openvino_docs_OV_UG_lpt_MarkupPrecisions>`
4. :doc:`MarkupPerTensorQuantization <openvino_docs_OV_UG_lpt_MarkupPerTensorQuantization>`
5. :doc:`MarkupAvgPoolPrecisionPreserved <openvino_docs_OV_UG_lpt_MarkupAvgPoolPrecisionPreserved>`
6. :doc:`PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>`
7. :doc:`AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>`
8. :doc:`AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>`
2. :doc:`MarkupCanBeQuantized <openvino_docs_OV_UG_lpt_MarkupCanBeQuantized>`
3. :doc:`MarkupPrecisions <openvino_docs_OV_UG_lpt_MarkupPrecisions>`
4. :doc:`MarkupPerTensorQuantization <openvino_docs_OV_UG_lpt_MarkupPerTensorQuantization>`
5. :doc:`MarkupAvgPoolPrecisionPreserved <openvino_docs_OV_UG_lpt_MarkupAvgPoolPrecisionPreserved>`
6. :doc:`PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>`
7. :doc:`AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>`
8. :doc:`AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>`
.. list-table::
:header-rows: 1
@ -47,16 +47,16 @@ This step defines the optimal ``FakeQuantize`` decomposition precisions for the
- Use attributes
* - MarkupBias
- Bias
-
-
* - MarkupCanBeQuantized
- Precisions
-
-
* - MarkupPrecisions
- Precisions,PrecisionPreserved
-
-
* - MarkupPerTensorQuantization
- PerTensorQuantization
-
-
* - MarkupAvgPoolPrecisionPreserved
- AvgPoolPrecisionPreserved
- Precisions, PrecisionPreserved
@ -70,16 +70,16 @@ This step defines the optimal ``FakeQuantize`` decomposition precisions for the
- QuantizationAlignment
- PrecisionPreserved, PerTensorQuantization
.. note::
.. note::
The same type of attribute instances can be created in different transformations. This approach is the result of the transformation single-responsibility principle. For example, ``Precision`` attribute instances are created in ``MarkupCanBeQuantized`` and ``MarkupPrecisions`` transformations, but the reasons for their creation are different
Common markup transformations can be decomposed into simpler utility markup transformations. The order of Markup utility transformations is not important:
* :doc:`CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>`
* :doc:`CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>`
* :doc:`PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>`
* :doc:`PropagateToInput <openvino_docs_OV_UG_lpt_PropagateToInput>`
* :doc:`UpdateSharedPrecisionPreserved <openvino_docs_OV_UG_lpt_UpdateSharedPrecisionPreserved>`
* :doc:`CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>`
* :doc:`CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>`
* :doc:`PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>`
* :doc:`PropagateToInput <openvino_docs_OV_UG_lpt_PropagateToInput>`
* :doc:`UpdateSharedPrecisionPreserved <openvino_docs_OV_UG_lpt_UpdateSharedPrecisionPreserved>`
Let's explore all transformations and their relations in detail, using one and the same model:
@ -90,12 +90,12 @@ The original model key features:
* The first ``concat1`` concatenation operation has not quantized ``convolution1`` consumer.
* The second ``concat2`` concatenation operation has quantized ``convolution2`` consumer with requirements:
* The second ``concat2`` concatenation operation has quantized ``convolution2`` consumer with requirements:
* support ``unsigned int8`` on activations,
* per-tensor quantization.
* Between the ``concat2`` concatenation operation and ``Convolution`` there is an ``AvgPool`` operation, which mathematically should return an ``f32`` tensor. But the ``MarkupAvgPoolPrecisionPreserved`` transformation is active. This allows the low precision transformation, that goes after the ``AvgPool``, to propagate low precision tensor to the next consumer.
* Between the ``concat2`` concatenation operation and ``Convolution`` there is an ``AvgPool`` operation, which mathematically should return an ``f32`` tensor. But the ``MarkupAvgPoolPrecisionPreserved`` transformation is active. This allows the low precision transformation, that goes after the ``AvgPool``, to propagate low precision tensor to the next consumer.
Transformations are run with the following parameters:
@ -127,8 +127,8 @@ Model display features (here and below):
The transformation is required and includes two tasks:
1. Mark operation input ports (create ``Precision`` attribute instance) by provided restrictions: input port index and required precisions. Restrictions are provided as input argument in ``:ref:`ov::pass::low_precision::LowPrecision <doxid-classov_1_1pass_1_1low__precision_1_1_low_precision>``` constructor.
2. Mark precision preserved operations.
1. Mark operation input ports (create ``Precision`` attribute instance) by provided restrictions: input port index and required precisions. Restrictions are provided as input argument in ``ov::pass::low_precision::LowPrecision`` constructor.
2. Mark precision preserved operations.
No attributes are required before the transformation. Changes in the example model after ``MarkupPrecisions`` transformation:
@ -191,7 +191,7 @@ Result model:
.. image:: _static/images/step2_markup5.svg
:alt: PropagatePrecisions
.. note::
.. note::
``AlignQuantizationIntervals`` and ``AlignQuantizationParameters`` transformations are required if the model has quantized concatenation operations.
6. AlignQuantizationIntervals

64
docs/articles_en/documentation/openvino_extensibility/ov_transformations.rst
View File

@ -5,7 +5,7 @@ Overview of Transformations API
.. meta::
:description: Learn how to apply additional model optimizations or transform
:description: Learn how to apply additional model optimizations or transform
unsupported subgraphs and operations, using OpenVINO™ Transformations API.
@ -17,19 +17,19 @@ Overview of Transformations API
openvino_docs_Extensibility_UG_matcher_pass
openvino_docs_Extensibility_UG_graph_rewrite_pass
OpenVINO Transformation mechanism allows to develop transformation passes to modify ``:ref:`ov::Model <doxid-classov_1_1_model>```. You can use this mechanism to apply additional optimizations to the original Model or transform unsupported subgraphs and operations to new operations which are supported by the plugin.
OpenVINO Transformation mechanism allows to develop transformation passes to modify ``ov::Model``. You can use this mechanism to apply additional optimizations to the original Model or transform unsupported subgraphs and operations to new operations which are supported by the plugin.
This guide contains all necessary information that you need to start implementing OpenVINO™ transformations.
Working with Model
##################
Before the moving to transformation part it is needed to say several words about functions which allow to modify ``:ref:`ov::Model <doxid-classov_1_1_model>```.
This chapter extends the :doc:`model representation guide <openvino_docs_OV_UG_Model_Representation>` and shows an API that allows us to manipulate with ``:ref:`ov::Model <doxid-classov_1_1_model>```.
Before the moving to transformation part it is needed to say several words about functions which allow to modify ``ov::Model``.
This chapter extends the :doc:`model representation guide <openvino_docs_OV_UG_Model_Representation>` and shows an API that allows us to manipulate with ``ov::Model``.
Working with node input and output ports
++++++++++++++++++++++++++++++++++++++++
First of all let's talk about ``:ref:`ov::Node <doxid-classov_1_1_node>``` input/output ports. Each OpenVINO™ operation has input and output ports except cases when operation has ``Parameter`` or ``Constant`` type.
First of all let's talk about ``ov::Node`` input/output ports. Each OpenVINO™ operation has input and output ports except cases when operation has ``Parameter`` or ``Constant`` type.
Every port belongs to its node, so using a port we can access parent node, get shape and type for particular input/output, get all consumers in case of output port, and get producer node in case of input port.
With output port we can set inputs for newly created operations.
@ -49,13 +49,13 @@ Let's start with OpenVINO™ helper functions. The most popular function is ``ov
We will review real replacement case where Negative operation is replaced with Multiply.
.. image:: ./_static/images/ngraph_replace_node.png
.. image:: ./_static/images/ngraph_replace_node.png
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
:fragment: [ov:replace_node]
``:ref:`ov::replace_node <doxid-namespaceov_1a75d84ee654edb73fe4fb18936a5dca6d>``` has a constraint that number of output ports for both of ops must be the same; otherwise, it raises an exception.
``ov::replace_node`` has a constraint that number of output ports for both of ops must be the same; otherwise, it raises an exception.
The alternative way to do the same replacement is the following:
@ -71,7 +71,7 @@ Another transformation example is insertion.
:language: cpp
:fragment: [ov:insert_node]
The alternative way to the insert operation is to make a node copy and use ``:ref:`ov::replace_node() <doxid-namespaceov_1a75d84ee654edb73fe4fb18936a5dca6d>```:
The alternative way to the insert operation is to make a node copy and use ``ov::replace_node()``:
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
@ -88,16 +88,16 @@ To eliminate operation, OpenVINO™ has special method that considers all limita
:language: cpp
:fragment: [ov:eliminate_node]
``:ref:`ov::replace_output_update_name() <doxid-namespaceov_1a75ba2120e573883bd96bb19c887c6a1d>``` in case of successful replacement it automatically preserves friendly name and runtime info.
``ov::replace_output_update_name()`` in case of successful replacement it automatically preserves friendly name and runtime info.
.. _transformations_types:
Transformations types
Transformations types
#####################
OpenVINO™ Runtime has three main transformation types:
* :doc:`Model pass <openvino_docs_Extensibility_UG_model_pass>` - straightforward way to work with ``:ref:`ov::Model <doxid-classov_1_1_model>``` directly
* :doc:`Model pass <openvino_docs_Extensibility_UG_model_pass>` - straightforward way to work with ``ov::Model`` directly
* :doc:`Matcher pass <openvino_docs_Extensibility_UG_matcher_pass>` - pattern-based transformation approach
* :doc:`Graph rewrite pass <openvino_docs_Extensibility_UG_graph_rewrite_pass>` - container for matcher passes needed for efficient execution
@ -108,12 +108,12 @@ Transformation conditional compilation
Transformation library has two internal macros to support conditional compilation feature.
* ``:ref:`MATCHER_SCOPE(region) <doxid-conditional__compilation_2include_2openvino_2cc_2pass_2itt_8hpp_1a3d1377542bcf3e305c33a1b683cc77df>``` - allows to disable the MatcherPass if matcher isn't used. The region name should be unique. This macro creates a local variable ``matcher_name`` which you should use as a matcher name.
* ``:ref:`RUN_ON_MODEL_SCOPE(region) <doxid-conditional__compilation_2include_2openvino_2cc_2pass_2itt_8hpp_1ab308561b849d47b9c820506ec73c4a30>``` - allows to disable run_on_model pass if it isn't used. The region name should be unique.
* ``MATCHER_SCOPE(region)`` - allows to disable the MatcherPass if matcher isn't used. The region name should be unique. This macro creates a local variable ``matcher_name`` which you should use as a matcher name.
* ``RUN_ON_MODEL_SCOPE(region)`` - allows to disable run_on_model pass if it isn't used. The region name should be unique.
.. _transformation_writing_essentials:
Transformation writing essentials
Transformation writing essentials
#################################
When developing a transformation, you need to follow these transformation rules:
@ -121,7 +121,7 @@ When developing a transformation, you need to follow these transformation rules:
1. Friendly Names
+++++++++++++++++
Each ``:ref:`ov::Node <doxid-classov_1_1_node>``` has an unique name and a friendly name. In transformations we care only about friendly name because it represents the name from the model.
Each ``ov::Node`` has an unique name and a friendly name. In transformations we care only about friendly name because it represents the name from the model.
To avoid losing friendly name when replacing node with other node or subgraph, set the original friendly name to the latest node in replacing subgraph. See the example below.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
@ -133,8 +133,8 @@ In more advanced cases, when replaced operation has several outputs and we add a
2. Runtime Info
+++++++++++++++
Runtime info is a map ``std::map<std::string, :ref:`ov::Any <doxid-classov_1_1_any>`>`` located inside ``:ref:`ov::Node <doxid-classov_1_1_node>``` class. It represents additional attributes in ``:ref:`ov::Node <doxid-classov_1_1_node>```.
These attributes can be set by users or by plugins and when executing transformation that changes ``:ref:`ov::Model <doxid-classov_1_1_model>``` we need to preserve these attributes as they will not be automatically propagated.
Runtime info is a map ``std::map<std::string, ov::Any>`` located inside ``ov::Node`` class. It represents additional attributes in ``ov::Node>``.
These attributes can be set by users or by plugins and when executing transformation that changes ``ov::Model`` we need to preserve these attributes as they will not be automatically propagated.
In most cases, transformations have the following types: 1:1 (replace node with another node), 1:N (replace node with a sub-graph), N:1 (fuse sub-graph into a single node), N:M (any other transformation).
Currently, there is no mechanism that automatically detects transformation types, so we need to propagate this runtime information manually. See the examples below.
@ -143,21 +143,21 @@ Currently, there is no mechanism that automatically detects transformation types
:language: cpp
:fragment: [ov:copy_runtime_info]
When transformation has multiple fusions or decompositions, ``:ref:`ov::copy_runtime_info <doxid-namespaceov_1a3bb5969a95703b4b4fd77f6f58837207>``` must be called multiple times for each case.
When transformation has multiple fusions or decompositions, ``ov::copy_runtime_info`` must be called multiple times for each case.
.. note:: ``copy_runtime_info`` removes ``rt_info`` from destination nodes. If you want to keep it, you need to specify them in source nodes like this: ``copy_runtime_info({a, b, c}, {a, b})``
3. Constant Folding
+++++++++++++++++++
If your transformation inserts constant sub-graphs that need to be folded, do not forget to use ``:ref:`ov::pass::ConstantFolding() <doxid-classov_1_1pass_1_1_constant_folding>``` after your transformation or call constant folding directly for operation.
If your transformation inserts constant sub-graphs that need to be folded, do not forget to use ``ov::pass::ConstantFolding()`` after your transformation or call constant folding directly for operation.
The example below shows how constant subgraph can be constructed.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
:fragment: [ov:constant_subgraph]
Manual constant folding is more preferable than ``:ref:`ov::pass::ConstantFolding() <doxid-classov_1_1pass_1_1_constant_folding>``` because it is much faster.
Manual constant folding is more preferable than ``ov::pass::ConstantFolding()`` because it is much faster.
Below you can find an example of manual constant folding:
@ -167,28 +167,28 @@ Below you can find an example of manual constant folding:
.. _common_mistakes:
Common mistakes in transformations
Common mistakes in transformations
##################################
In transformation development process:
* Do not use deprecated OpenVINO™ API. Deprecated methods has the ``OPENVINO_DEPRECATED`` macros in its definition.
* Do not pass ``shared_ptr<Node>`` as an input for other node if type of node is unknown or it has multiple outputs. Use explicit output port.
* If you replace node with another node that produces different shape, remember that new shape will not be propagated until the first ``validate_nodes_and_infer_types`` call for ``:ref:`ov::Model <doxid-classov_1_1_model>```. If you are using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```, it will automatically call this method after each transformation execution.
* Do not forget to call the ``:ref:`ov::pass::ConstantFolding <doxid-classov_1_1pass_1_1_constant_folding>``` pass if your transformation creates constant subgraphs.
* If you replace node with another node that produces different shape, remember that new shape will not be propagated until the first ``validate_nodes_and_infer_types`` call for ``ov::Model``. If you are using ``ov::pass::Manager``, it will automatically call this method after each transformation execution.
* Do not forget to call the ``ov::pass::ConstantFolding`` pass if your transformation creates constant subgraphs.
* Use latest OpSet if you are not developing downgrade transformation pass.
* When developing a callback for ``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>```, do not change nodes that come after the root node in topological order.
* When developing a callback for ``ov::pass::MatcherPass``, do not change nodes that come after the root node in topological order.
.. _using_pass_manager:
Using pass manager
##################
``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` is a container class that can store the list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
``ov::pass::Manager`` is a container class that can store the list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
It can register and apply any `transformation pass <#transformations_types>`__ on model.
In addition, ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` has extended debug capabilities (find more information in the `how to debug transformations <#how_to_debug_transformations>`__ section).
In addition, ``ov::pass::Manager`` has extended debug capabilities (find more information in the `how to debug transformations <#how_to_debug_transformations>`__ section).
The example below shows basic usage of ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```
The example below shows basic usage of ``ov::pass::Manager``
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
@ -199,16 +199,16 @@ Another example shows how multiple matcher passes can be united into single Grap
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
:fragment: [matcher_pass:manager2]
.. _how_to_debug_transformations:
How to debug transformations
.. _how_to_debug_transformations:
How to debug transformations
############################
If you are using ``ngraph::pass::Manager`` to run sequence of transformations, you can get additional debug capabilities by using the following environment variables:
.. code-block:: cpp
OV_PROFILE_PASS_ENABLE=1 - enables performance measurement for each transformation and prints execution status
OV_ENABLE_VISUALIZE_TRACING=1 - enables visualization after each transformation. By default, it saves dot and svg files.
@ -218,6 +218,6 @@ If you are using ``ngraph::pass::Manager`` to run sequence of transformations, y
See Also
########
* :doc:`OpenVINO™ Model Representation <openvino_docs_OV_UG_Model_Representation>`
* :doc:`OpenVINO™ Model Representation <openvino_docs_OV_UG_Model_Representation>`
* :doc:`OpenVINO™ Extensions <openvino_docs_Extensibility_UG_Intro>`

10
docs/articles_en/documentation/openvino_extensibility/ov_transformations/graph_rewrite_pass.rst
View File

@ -5,11 +5,11 @@ OpenVINO Graph Rewrite Pass
.. meta::
:description: Get to know how Graph Rewrite handles running multiple matcher passes on
:description: Get to know how Graph Rewrite handles running multiple matcher passes on
ov::Model in a single graph traversal.
``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>``` serves for running multiple matcher passes on ``:ref:`ov::Model <doxid-classov_1_1_model>``` in a single graph traversal.
``ov::pass::GraphRewrite`` serves for running multiple matcher passes on ``ov::Model`` in a single graph traversal.
Example:
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
@ -18,13 +18,13 @@ Example:
In addition, GraphRewrite handles nodes that were registered by MatcherPasses during their execution. This nodes will be added to the beginning of the sequence with nodes for pattern matching.
.. note::
.. note::
When using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` temporary GraphRewrite is used to execute single MatcherPass.
When using ``ov::pass::Manager`` temporary GraphRewrite is used to execute single MatcherPass.
GraphRewrite has two algorithms for MatcherPasses execution. First algorithm is straightforward. It applies each MatcherPass in registration order to current node.
.. image:: ./_static/images/graph_rewrite_execution.png
.. image:: ./_static/images/graph_rewrite_execution.png
But it is not really efficient when you have a lot of registered passes. So first of all GraphRewrite checks that all MatcherPass patterns has type-based root node (it means that type of this node is not hidden into predicate).
And then creates map from registered MatcherPasses. That helps to avoid additional cost of applying each MatcherPass for each node.

32
docs/articles_en/documentation/openvino_extensibility/ov_transformations/matcher_pass.rst
View File

@ -5,11 +5,11 @@ OpenVINO Matcher Pass
.. meta::
:description: Learn how to create a pattern, implement a callback, register
the pattern and Matcher to execute MatcherPass transformation
:description: Learn how to create a pattern, implement a callback, register
the pattern and Matcher to execute MatcherPass transformation
on a model.
``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>``` is used for pattern-based transformations.
``ov::pass::MatcherPass`` is used for pattern-based transformations.
Template for MatcherPass transformation class
@ -22,7 +22,7 @@ Template for MatcherPass transformation class
:fragment: [graph_rewrite:template_transformation_cpp]
To use ``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>```, you need to complete these steps:
To use ``ov::pass::MatcherPass``, you need to complete these steps:
1. Create a pattern
2. Implement a callback
@ -34,10 +34,10 @@ So let's go through each of these steps.
Create a pattern
################
Pattern is a single root ``:ref:`ov::Model <doxid-classov_1_1_model>```. But the only difference is that you do not need to create a model object, you just need to create and connect opset or special pattern operations.
Pattern is a single root ``ov::Model``. But the only difference is that you do not need to create a model object, you just need to create and connect opset or special pattern operations.
Then you need to take the last created operation and put it as a root of the pattern. This root node will be used as a root node in pattern matching.
.. note::
.. note::
Any nodes in a pattern that have no consumers and are not registered as root will not be used in pattern matching.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
@ -60,13 +60,13 @@ Callback is an action applied to every pattern entrance. In general, callback is
The example above shows the callback structure and how Matcher can be used for accessing nodes detected by pattern.
Callback return value is ``true`` if root node was replaced and another pattern cannot be applied to the same root node; otherwise, it is ``false``.
.. note::
.. note::
It is not recommended to manipulate with nodes that are under root node. This may affect GraphRewrite execution as it is expected that all nodes that come after root node in topological order are valid and can be used in pattern matching.
MatcherPass also provides functionality that allows reporting of the newly created nodes that can be used in additional pattern matching.
If MatcherPass was registered in ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` or ``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>```, these registered nodes will be added for additional pattern matching.
That means that matcher passes registered in ``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>``` will be applied to these nodes.
If MatcherPass was registered in ``ov::pass::Manager`` or ``ov::pass::GraphRewrite``, these registered nodes will be added for additional pattern matching.
That means that matcher passes registered in ``ov::pass::GraphRewrite`` will be applied to these nodes.
The example below shows how single MatcherPass can fuse sequence of operations using the ``register_new_node`` method.
@ -74,7 +74,7 @@ The example below shows how single MatcherPass can fuse sequence of operations u
:language: cpp
:fragment: [matcher_pass:relu_fusion]
.. note::
.. note::
If you register multiple nodes, please add them in topological order. We do not topologically sort these nodes as it is a time-consuming operation.
Register pattern and Matcher
@ -82,12 +82,12 @@ Register pattern and Matcher
The last step is to register Matcher and callback inside the MatcherPass pass. To do this, call the ``register_matcher`` method.
.. note::
.. note::
Only one matcher can be registered for a single MatcherPass class.
.. code-block:: cpp
// Register matcher and callback
register_matcher(m, callback);
@ -103,13 +103,13 @@ MatcherPass has multiple ways to be executed:
:language: cpp
:fragment: [matcher_pass:run_on_node]
* Run on ``:ref:`ov::Model <doxid-classov_1_1_model>``` using GraphRewrite - this approach gives ability to run MatcherPass on whole ``:ref:`ov::Model <doxid-classov_1_1_model>```. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
* Run on ``ov::Model`` using GraphRewrite - this approach gives ability to run MatcherPass on whole ``ov::Model``. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
:fragment: [matcher_pass:graph_rewrite]
* Run on ``:ref:`ov::Model <doxid-classov_1_1_model>``` using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` - this approach helps you to register MatcherPass for execution on ``:ref:`ov::Model <doxid-classov_1_1_model>``` as another transformation types.
* Run on ``ov::Model`` using ``ov::pass::Manager`` - this approach helps you to register MatcherPass for execution on ``ov::Model`` as another transformation types.
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
@ -125,8 +125,8 @@ And for these cases OpenVINO™ provides additional helpers to construct pattern
There are two main helpers:
1. ``:ref:`ov::pass::pattern::any_input <doxid-namespaceov_1_1pass_1_1pattern_1a8ed84c3eed4610f117ee10d86d500e02>``` - helps to express inputs if their types are undefined.
2. ``:ref:`ov::pass::pattern::wrap_type <doxid-namespaceov_1_1pass_1_1pattern_1adfcd6031c95d7bace5f084e2aa105af8>`<T>`` - helps to express nodes of pattern without specifying node attributes.
1. ``ov::pass::pattern::any_input`` - helps to express inputs if their types are undefined.
2. ``ov::pass::pattern::wrap_type <T>`` - helps to express nodes of pattern without specifying node attributes.
Let's go through the example to have better understanding of how it works:

12
docs/articles_en/documentation/openvino_extensibility/ov_transformations/model_pass.rst
View File

@ -5,25 +5,25 @@ OpenVINO Model Pass
.. meta::
:description: Learn how to use Model Pass transformation class to take entire
:description: Learn how to use Model Pass transformation class to take entire
ov::Model as input and process it.
``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>``` is used for transformations that take entire ``:ref:`ov::Model <doxid-classov_1_1_model>``` as an input and process it.
``ov::pass::ModelPass`` is used for transformations that take entire ``ov::Model`` as an input and process it.
Template for ModelPass transformation class
.. doxygensnippet:: docs/snippets/template_model_transformation.hpp
:language: cpp
.. doxygensnippet:: docs/snippets/template_model_transformation.hpp
:language: cpp
:fragment: [model_pass:template_transformation_hpp]
.. doxygensnippet:: docs/snippets/template_model_transformation.cpp
:language: cpp
:fragment: [model_pass:template_transformation_cpp]
Using ``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>```, you need to override the ``run_on_model`` method where you will write the transformation code.
Using ``ov::pass::ModelPass``, you need to override the ``run_on_model`` method where you will write the transformation code.
Return value is ``true`` if the original model has changed during transformation (new operation was added, or operations replacement was made, or node attributes were changed); otherwise, it is ``false``.
Also ``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>``` based transformations can be executed via ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```.
Also ``ov::pass::ModelPass`` based transformations can be executed via ``ov::pass::Manager``.
See Also
########

154
docs/articles_en/openvino_workflow/running_inference_with_openvino/Device_Plugins/GPU/GPU_RemoteTensor_API.rst
View File

@ -5,13 +5,13 @@ Remote Tensor API of GPU Plugin
.. meta::
:description: The Remote Tensor API of GPU plugin in OpenVINO™ supports
interoperability with existing native APIs, such as OpenCL,
:description: The Remote Tensor API of GPU plugin in OpenVINO™ supports
interoperability with existing native APIs, such as OpenCL,
Microsoft DirectX, or VAAPI.
The GPU plugin implementation of the ``ov::RemoteContext`` and ``ov::RemoteTensor`` interfaces supports GPU
pipeline developers who need video memory sharing and interoperability with existing native APIs,
pipeline developers who need video memory sharing and interoperability with existing native APIs,
such as OpenCL, Microsoft DirectX, or VAAPI.
The ``ov::RemoteContext`` and ``ov::RemoteTensor`` interface implementation targets the need for memory sharing and
@ -57,24 +57,24 @@ of ``ov::RemoteContext`` derived classes.
:sync: windows-cpp
.. tab-set::
.. tab-item:: Create from cl_context
:sync: create-from-cl-context
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_cl_context]
.. tab-item:: Create from cl_queue
:sync: create-from-cl-queue
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_cl_queue]
.. tab-item:: Create from ID3D11Device
:sync: create-from-id3d11device
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_d3d_device]
@ -83,24 +83,24 @@ of ``ov::RemoteContext`` derived classes.
:sync: windows-c
.. tab-set::
.. tab-item:: Create from cl_context
:sync: create-from-cl-context
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_cl_context]
.. tab-item:: Create from cl_queue
:sync: create-from-cl-queue
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_cl_queue]
.. tab-item:: Create from ID3D11Device
:sync: create-from-id3d11device
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_d3d_device]
@ -109,54 +109,54 @@ of ``ov::RemoteContext`` derived classes.
:sync: linux-cpp
.. tab-set::
.. tab-item:: Create from cl_context
:sync: create-from-cl-context
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_cl_context]
.. tab-item:: Create from cl_queue
:sync: create-from-cl-queue
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_cl_queue]
.. tab-item:: Create from VADisplay
:sync: create-from-vadisplay
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [context_from_va_display]
.. tab-item:: Linux/C
:sync: linux-c
.. tab-set::
.. tab-item:: Create from cl_context
:sync: create-from-cl-context
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_cl_context]
.. tab-item:: Create from cl_queue
:sync: create-from-cl-queue
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_cl_queue]
.. tab-item:: Create from VADisplay
:sync: create-from-vadisplay
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [context_from_va_display]
Getting RemoteContext from the Plugin
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@ -173,40 +173,40 @@ To request the current default context of the plugin, use one of the following m
:sync: cpp
.. tab-set::
.. tab-item:: Get context from Core
:sync: get-context-core
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [default_context_from_core]
.. tab-item:: Get context from compiled model
:sync: get-context-compiled-model
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp
:language: cpp
:fragment: [default_context_from_model]
.. tab-item:: C
:sync: c
.. tab-set::
.. tab-item:: Get context from Core
:sync: get-context-core
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [default_context_from_core]
.. tab-item:: Get context from compiled model
:sync: get-context-compiled-model
.. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp
:language: c
:fragment: [default_context_from_model]
Memory Sharing Between Application and GPU Plugin
###########################################################
@ -359,11 +359,11 @@ Direct NV12 Video Surface Input
To support the direct consumption of a hardware video decoder output, the GPU plugin accepts:
* Two-plane NV12 video surface input - calling the ``create_tensor_nv12()`` function creates
a pair of ``ov::RemoteTensor`` objects, representing the Y and UV planes.
* Single-plane NV12 video surface input - calling the ``create_tensor()`` function creates one
* Two-plane NV12 video surface input - calling the ``create_tensor_nv12()`` function creates
a pair of ``ov::RemoteTensor`` objects, representing the Y and UV planes.
* Single-plane NV12 video surface input - calling the ``create_tensor()`` function creates one
``ov::RemoteTensor`` object, representing the Y and UV planes at once (Y elements before UV elements).
* NV12 to Grey video surface input conversion - calling the ``create_tensor()`` function creates one
* NV12 to Grey video surface input conversion - calling the ``create_tensor()`` function creates one
``ov::RemoteTensor`` object, representing only the Y plane.
To ensure that the plugin generates a correct execution graph, static preprocessing
@ -375,102 +375,102 @@ should be added before model compilation:
:sync: two-plane
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes.cpp
:language: cpp
:fragment: [init_preproc]
.. tab-item:: C
:sync: c
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes_c.cpp
:language: c
:fragment: [init_preproc]
.. tab-item:: single-plane
:sync: single-plane
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_single_plane.cpp
:language: cpp
:fragment: [init_preproc]
.. tab-item:: NV12 to Grey
:sync: nv12-grey
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_to_gray.cpp
:language: cpp
:fragment: [init_preproc]
Since the ``ov::intel_gpu::ocl::ClImage2DTensor`` and its derived classes do not support batched surfaces,
if batching and surface sharing are required at the same time,
Since the ``ov::intel_gpu::ocl::ClImage2DTensor`` and its derived classes do not support batched surfaces,
if batching and surface sharing are required at the same time,
inputs need to be set via the ``ov::InferRequest::set_tensors`` method with vector of shared surfaces for each plane:
.. tab-set::
.. tab-item:: Single Batch
:sync: single-batch
.. tab-set::
.. tab-item:: two-plane
:sync: two-plane
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes.cpp
:language: cpp
:fragment: [single_batch]
.. tab-item:: C
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes_c.cpp
:language: c
:fragment: [single_batch]
.. tab-item:: single-plane
:sync: single-plane
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_single_plane.cpp
:language: cpp
:fragment: [single_batch]
.. tab-item:: NV12 to Grey
:sync: nv12-grey
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_to_gray.cpp
:language: cpp
:fragment: [single_batch]
.. tab-item:: Multiple Batches
:sync: multiple-batches
.. tab-set::
.. tab-item:: two-plane
:sync: two-plane
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes.cpp
:language: cpp
:fragment: [batched_case]
.. tab-item:: single-plane
:sync: single-plane
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_single_plane.cpp
:language: cpp
:fragment: [batched_case]
.. tab-item:: NV12 to Grey
:sync: nv12-grey
.. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_to_gray.cpp
:language: cpp
:fragment: [batched_case]
@ -524,7 +524,7 @@ To see pseudo-code of usage examples, refer to the sections below.
.. NOTE::
For low-level parameter usage examples, see the source code of user-side wrappers from the include files mentioned above.
@ -543,19 +543,19 @@ To see pseudo-code of usage examples, refer to the sections below.
:fragment: [context_sharing_user_handle]
.. dropdown:: Direct Consuming of the NV12 VAAPI Video Decoder Surface on Linux
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/context_sharing_va.cpp
:language: cpp
:fragment: [context_sharing_va]
.. tab-item:: C
:sync: c
.. doxygensnippet:: docs/snippets/gpu/context_sharing_va_c.cpp
:language: c
:fragment: [context_sharing_va]
@ -563,6 +563,6 @@ To see pseudo-code of usage examples, refer to the sections below.
See Also
#######################################
* ``:ref:`ov::Core <doxid-classov-1-1-core>```
* ``:ref:`ov::RemoteTensor <doxid-classov-1-1-remote-tensor>```
* ``ov::Core``
* ``ov::RemoteTensor``

14
docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst
View File

@ -78,9 +78,9 @@ Example of applying LowLatency2 transformation:
.. image:: _static/images/applying_low_latency_2.svg
:align: center
After applying the transformation, ReadValue operations can receive other operations as an input, as shown in the picture above.
These inputs should set the initial value for initialization of ReadValue operations.
However, such initialization is not supported in the current State API implementation.
After applying the transformation, ReadValue operations can receive other operations as an input, as shown in the picture above.
These inputs should set the initial value for initialization of ReadValue operations.
However, such initialization is not supported in the current State API implementation.
Input values are ignored and the initial values for the ReadValue operations are set to zeros unless otherwise specified
by the user via :ref:`State API<ov_ug_state_api>`.
@ -180,7 +180,7 @@ How to get TensorIterator/Loop operations from different frameworks via ModelOpt
How to create a model with state using OpenVINO
###############################################
To get a model with states ready for inference, you can convert a model from another framework to IR with Model Optimizer
To get a model with states ready for inference, you can convert a model from another framework to IR with Model Optimizer
or create an OpenVINO Model (details can be found in :doc:`Build OpenVINO Model section<openvino_docs_OV_UG_Model_Representation>`.
Let's build the following model using C++ OpenVINO API:
@ -194,8 +194,8 @@ Example of Creating Model via OpenVINO API
.. doxygensnippet:: docs/snippets/ov_stateful_models_intro.cpp
:language: cpp
:fragment: [ov:state_model]
:fragment: [ov:state_network]
In this example, `ov::SinkVector` is used to create `ov::Model`. For model with states, except inputs and outputs, `Assign` nodes should also point to `Model`
to avoid deleting it during graph transformations. You can do it with the constructor, as shown in the example, or with the special method `add_sinks(const SinkVector& sinks)`. Also, you can delete
In this example, `ov::SinkVector` is used to create `ov::Model`. For model with states, except inputs and outputs, `Assign` nodes should also point to `Model`
to avoid deleting it during graph transformations. You can do it with the constructor, as shown in the example, or with the special method `add_sinks(const SinkVector& sinks)`. Also, you can delete
sink from `ov::Model` after deleting the node from graph with the `delete_sink()` method.