diff --git a/docs/Extensibility_UG/GPU_Extensibility.md b/docs/Extensibility_UG/GPU_Extensibility.md index ae830870cff..0ef611e3fcf 100644 --- a/docs/Extensibility_UG/GPU_Extensibility.md +++ b/docs/Extensibility_UG/GPU_Extensibility.md @@ -18,18 +18,20 @@ There are two options for using the custom operation configuration file: .. tab-set:: - .. tab-item:: C++ - - .. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.cpp - :language: cpp - :fragment: [part0] - .. tab-item:: Python + :sync: py .. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.py :language: python :fragment: [part0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.cpp + :language: cpp + :fragment: [part0] + All OpenVINO samples, except the trivial ``hello_classification``, and most Open Model Zoo demos feature a dedicated command-line option ``-c`` to load custom kernels. For example, to load custom operations for the classification sample, run the command below: diff --git a/docs/Extensibility_UG/Intro.md b/docs/Extensibility_UG/Intro.md index ba34e4d98a0..61300e14b27 100644 --- a/docs/Extensibility_UG/Intro.md +++ b/docs/Extensibility_UG/Intro.md @@ -95,6 +95,13 @@ Extensions can be loaded from a code with the ``:ref:`ov::Core::add_extension < .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_extensions.py + :language: python + :fragment: [add_extension] + .. tab-item:: C++ :sync: cpp @@ -102,18 +109,18 @@ Extensions can be loaded from a code with the ``:ref:`ov::Core::add_extension < :language: cpp :fragment: [add_extension] - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/ov_extensions.py - :language: python - :fragment: [add_extension] - The ``Identity`` is a custom operation class defined in :doc:`Custom Operation Guide `. This is sufficient to enable reading OpenVINO IR which uses the ``Identity`` extension operation emitted by Model Optimizer. In order to load original model directly to the runtime, add a mapping extension: .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_extensions.py + :language: python + :fragment: [add_frontend_extension] + .. tab-item:: C++ :sync: cpp @@ -121,13 +128,6 @@ The ``Identity`` is a custom operation class defined in :doc:`Custom Operation G :language: cpp :fragment: [add_frontend_extension] - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/ov_extensions.py - :language: python - :fragment: [add_frontend_extension] - When Python API is used, there is no way to implement a custom OpenVINO operation. Even if custom OpenVINO operation is implemented in C++ and loaded into the runtime by a shared library, there is still no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping. Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used. @@ -177,13 +177,6 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_extensions.cpp - :language: cpp - :fragment: [add_extension_lib] - .. tab-item:: Python :sync: py @@ -191,6 +184,13 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command. :language: python :fragment: [add_extension_lib] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [add_extension_lib] + See Also ######## diff --git a/docs/Extensibility_UG/frontend_extensions.md b/docs/Extensibility_UG/frontend_extensions.md index 3a41031fe24..2190d58ed0e 100644 --- a/docs/Extensibility_UG/frontend_extensions.md +++ b/docs/Extensibility_UG/frontend_extensions.md @@ -94,13 +94,6 @@ In this case, you can directly say that 'MyRelu' -> ``Relu`` mapping should be u .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_extensions.cpp - :language: cpp - :fragment: [frontend_extension_MyRelu] - .. tab-item:: Python :sync: py @@ -108,6 +101,13 @@ In this case, you can directly say that 'MyRelu' -> ``Relu`` mapping should be u :language: python :fragment: [py_frontend_extension_MyRelu] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_MyRelu] + In the resulting converted OpenVINO model, “MyRelu” operation will be replaced by the standard operation ``Relu`` from the latest available OpenVINO operation set. Notice that when standard operation is used, @@ -335,13 +335,6 @@ from ONNX according to the formula: ``ThresholdedRelu(x, alpha) -> Multiply(x, C .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_extensions.cpp - :language: cpp - :fragment: [frontend_extension_ThresholdedReLU_header] - .. tab-item:: Python :sync: py @@ -349,14 +342,14 @@ from ONNX according to the formula: ``ThresholdedRelu(x, alpha) -> Multiply(x, C :language: python :fragment: [py_frontend_extension_ThresholdedReLU_header] -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_extensions.cpp :language: cpp - :fragment: [frontend_extension_ThresholdedReLU] + :fragment: [frontend_extension_ThresholdedReLU_header] + +.. tab-set:: .. tab-item:: Python :sync: py @@ -365,6 +358,13 @@ from ONNX according to the formula: ``ThresholdedRelu(x, alpha) -> Multiply(x, C :language: python :fragment: [py_frontend_extension_ThresholdedReLU] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_ThresholdedReLU] + The next example shows how to use ``ConversionExtension`` to convert PyTorch `aten::hardtanh `_ diff --git a/docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md b/docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md index edb9e8385ac..56357ab9933 100644 --- a/docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md +++ b/docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md @@ -26,7 +26,7 @@ To convert a model to OpenVINO model format (``ov.Model``), you can use the foll .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -34,7 +34,7 @@ To convert a model to OpenVINO model format (``ov.Model``), you can use the foll ov_model = convert_model(INPUT_MODEL) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -61,7 +61,7 @@ To get the full list of conversion parameters, run the following command: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -69,7 +69,7 @@ To get the full list of conversion parameters, run the following command: ov_model = convert_model(help=True) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -86,7 +86,7 @@ Below is a list of separate examples for different frameworks and model conversi .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -94,7 +94,7 @@ Below is a list of separate examples for different frameworks and model conversi ov_model = convert_model("MobileNet.pb") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -106,7 +106,7 @@ Below is a list of separate examples for different frameworks and model conversi .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -114,7 +114,7 @@ Below is a list of separate examples for different frameworks and model conversi ov_model = convert_model("BERT", input_shape=[[2,30],[2,30],[2,30]]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -128,7 +128,7 @@ Below is a list of separate examples for different frameworks and model conversi .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -136,7 +136,7 @@ Below is a list of separate examples for different frameworks and model conversi ov_model = convert_model("ocr.onnx", output="probabilities") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -154,7 +154,7 @@ Below is a list of separate examples for different frameworks and model conversi .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -162,7 +162,7 @@ Below is a list of separate examples for different frameworks and model conversi ov_model = convert_model("unet.pdmodel", mean_values=[123,117,104], scale=255) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/MO_DG/prepare_model/Additional_Optimizations.md b/docs/MO_DG/prepare_model/Additional_Optimizations.md index 19c831adf73..4b30daaa18b 100644 --- a/docs/MO_DG/prepare_model/Additional_Optimizations.md +++ b/docs/MO_DG/prepare_model/Additional_Optimizations.md @@ -34,7 +34,7 @@ For example, the following command specifies the ``NHWC`` layout for a Tensorflo .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -42,7 +42,7 @@ For example, the following command specifies the ``NHWC`` layout for a Tensorflo ov_model = convert_model("tf_nasnet_large.onnx", layout="nhwc") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -59,7 +59,7 @@ having two dimensions: batch and size of the image expressed as the ``N?`` layou .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -67,7 +67,7 @@ having two dimensions: batch and size of the image expressed as the ``N?`` layou ov_model = convert_model("yolov3-tiny.onnx", layout={"input_1": "nchw", "image_shape": "n?"}) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -87,7 +87,7 @@ the following commands to provide data in the ``NCHW`` layout: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -97,7 +97,7 @@ the following commands to provide data in the ``NCHW`` layout: ov_model = convert_model("tf_nasnet_large.onnx", layout="nhwc->nchw") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -115,7 +115,7 @@ mentioned earlier, use the following commands: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -125,7 +125,7 @@ mentioned earlier, use the following commands: ov_model = convert_model("yolov3-tiny.onnx", layout={"input_1": "nchw->nhwc", "image_shape": "n?"} .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -168,7 +168,7 @@ model and applies mean-scale normalization to the input data: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -176,7 +176,7 @@ model and applies mean-scale normalization to the input data: ov_model = convert_model("unet.pdmodel", mean_values=[123,117,104], scale=255) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -211,7 +211,7 @@ model and embeds the ``reverse_input_channel`` preprocessing block into OpenVINO .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -219,7 +219,7 @@ model and embeds the ``reverse_input_channel`` preprocessing block into OpenVINO ov_model = convert_model("alexnet.pb", reverse_input_channels=True) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/MO_DG/prepare_model/FP16_Compression.md b/docs/MO_DG/prepare_model/FP16_Compression.md index 573d7c1d02d..44a829ef501 100644 --- a/docs/MO_DG/prepare_model/FP16_Compression.md +++ b/docs/MO_DG/prepare_model/FP16_Compression.md @@ -12,7 +12,7 @@ To compress the model, use the ``compress_to_fp16=True`` option: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -20,7 +20,7 @@ To compress the model, use the ``compress_to_fp16=True`` option: ov_model = convert_model(INPUT_MODEL, compress_to_fp16=False) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/MO_DG/prepare_model/convert_model/Converting_Model.md b/docs/MO_DG/prepare_model/convert_model/Converting_Model.md index 664b5508317..5decb389ab2 100644 --- a/docs/MO_DG/prepare_model/convert_model/Converting_Model.md +++ b/docs/MO_DG/prepare_model/convert_model/Converting_Model.md @@ -31,7 +31,7 @@ and specify the input shape of ``[2,300,300,3]``: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -39,7 +39,7 @@ and specify the input shape of ``[2,300,300,3]``: ov_model = convert_model("MobileNet.pb", input_shape=[2,300,300,3]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -54,7 +54,7 @@ and specify shapes ``[3,150,200,1]`` and ``[3]`` for them: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -62,7 +62,7 @@ and specify shapes ``[3,150,200,1]`` and ``[3]`` for them: ov_model = convert_model("ocr.onnx", input=["data","seq_len"], input_shape=[[3,150,200,1],[3]]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -74,7 +74,7 @@ Alternatively, specify input shapes, using the ``input`` parameter as follows: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -82,7 +82,7 @@ Alternatively, specify input shapes, using the ``input`` parameter as follows: ov_model = convert_model("ocr.onnx", input=[("data",[3,150,200,1]),("seq_len",[3])]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -97,7 +97,7 @@ For example, launch model conversion for the ONNX OCR model and specify dynamic .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -105,7 +105,7 @@ For example, launch model conversion for the ONNX OCR model and specify dynamic ov_model = convert_model("ocr.onnx", input=["data","seq_len"], input_shape=[[-1,150,200,1],[-1]] .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -119,7 +119,7 @@ For example, launch model conversion for the ONNX OCR model and specify a bounda .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -128,7 +128,7 @@ For example, launch model conversion for the ONNX OCR model and specify a bounda ov_model = convert_model("ocr.onnx", input=["data","seq_len"], input_shape=[[Dimension(1,3),150,200,1],[Dimension(1,3)]] .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/MO_DG/prepare_model/convert_model/Cutting_Model.md b/docs/MO_DG/prepare_model/convert_model/Cutting_Model.md index 55c60fe1187..4d10c0ddaad 100644 --- a/docs/MO_DG/prepare_model/convert_model/Cutting_Model.md +++ b/docs/MO_DG/prepare_model/convert_model/Cutting_Model.md @@ -53,7 +53,7 @@ Convert this model to ``ov.Model``: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -61,7 +61,7 @@ Convert this model to ``ov.Model``: ov_model = convert_model("inception_v1.pb", batch=1) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -115,7 +115,7 @@ Due to automatic identification of inputs and outputs, providing the ``input`` a .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -125,7 +125,7 @@ Due to automatic identification of inputs and outputs, providing the ``input`` a ov_model = convert_model("inception_v1.pb", batch=1, input="input", output="InceptionV1/Logits/Predictions/Reshape_1") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -154,7 +154,7 @@ If you want to cut your model at the end, you have the following options: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -162,7 +162,7 @@ If you want to cut your model at the end, you have the following options: ov_model = convert_model("inception_v1.pb", batch=1, output="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -217,7 +217,7 @@ If you want to cut your model at the end, you have the following options: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -225,7 +225,7 @@ If you want to cut your model at the end, you have the following options: ov_model = convert_model("inception_v1.pb", batch=1, output="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu:0") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -280,7 +280,7 @@ If you want to cut your model at the end, you have the following options: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -288,7 +288,7 @@ If you want to cut your model at the end, you have the following options: ov_model = convert_model("inception_v1.pb", batch=1, output="0:InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -337,7 +337,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -345,7 +345,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` ov_model = convert_model("inception_v1.pb", batch=1, output="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu", input="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -388,7 +388,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -396,7 +396,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` ov_model = convert_model("inception_v1.pb", batch=1, input="0:InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu", output="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -439,7 +439,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -447,7 +447,7 @@ If you want to go further and cut the beginning of the model, leaving only the ` ov_model = convert_model("inception_v1.pb", batch=1, input="InceptionV1/InceptionV1/Conv2d_1a_7x7/BatchNorm/batchnorm/add_1:0", output="InceptionV1/InceptionV1/Conv2d_1a_7x7/Relu") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -494,7 +494,7 @@ Following this behavior, ``convert_model()`` creates an ``Input`` layer for port .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -502,7 +502,7 @@ Following this behavior, ``convert_model()`` creates an ``Input`` layer for port ov_model = convert_model("inception_v1.pb", batch=1, input="InceptionV1/InceptionV1/Conv2d_1a_7x7/convolution") .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -516,7 +516,7 @@ Different behavior occurs when ``input_shape`` is also used as an attempt to ove .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -524,7 +524,7 @@ Different behavior occurs when ``input_shape`` is also used as an attempt to ove ov_model = convert_model("inception_v1.pb", input="InceptionV1/InceptionV1/Conv2d_1a_7x7/convolution", input_shape=[1,224,224,3]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh @@ -546,7 +546,7 @@ The correct command line is: .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -554,7 +554,7 @@ The correct command line is: ov_model = convert_model("inception_v1.pb", input="0:InceptionV1/InceptionV1/Conv2d_1a_7x7/convolution", input_shape=[1,224,224,3]) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/OV_Runtime_UG/AutoPlugin_Debugging.md b/docs/OV_Runtime_UG/AutoPlugin_Debugging.md index e55cf362b45..97b483019fa 100644 --- a/docs/OV_Runtime_UG/AutoPlugin_Debugging.md +++ b/docs/OV_Runtime_UG/AutoPlugin_Debugging.md @@ -27,6 +27,13 @@ There are six levels of logs, which can be called explicitly or set via the ``OP .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_auto.py + :language: python + :fragment: [part6] + .. tab-item:: C++ :sync: cpp @@ -34,15 +41,8 @@ There are six levels of logs, which can be called explicitly or set via the ``OP :language: cpp :fragment: [part6] - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/ov_auto.py - :language: python - :fragment: [part6] - .. tab-item:: OS environment variable - :sync: os_env_variable + :sync: os-environment-variable .. code-block:: sh diff --git a/docs/OV_Runtime_UG/Model_caching_overview.md b/docs/OV_Runtime_UG/Model_caching_overview.md index 9b47de322e3..6d8af317cce 100644 --- a/docs/OV_Runtime_UG/Model_caching_overview.md +++ b/docs/OV_Runtime_UG/Model_caching_overview.md @@ -43,13 +43,6 @@ To enable model caching, the application must specify a folder to store the cach .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_caching.cpp - :language: cpp - :fragment: [ov:caching:part0] - .. tab-item:: Python :sync: py @@ -57,6 +50,13 @@ To enable model caching, the application must specify a folder to store the cach :language: py :fragment: [ov:caching:part0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_caching.cpp + :language: cpp + :fragment: [ov:caching:part0] + With this code, if the device specified by ``device_name`` supports import/export model capability, a cached blob is automatically created inside the ``/path/to/cache/dir`` folder. @@ -77,13 +77,6 @@ For these cases, there is a more convenient API to compile the model in a single .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_caching.cpp - :language: cpp - :fragment: [ov:caching:part1] - .. tab-item:: Python :sync: py @@ -91,18 +84,18 @@ For these cases, there is a more convenient API to compile the model in a single :language: py :fragment: [ov:caching:part1] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_caching.cpp + :language: cpp + :fragment: [ov:caching:part1] + With model caching enabled, the total load time is even shorter, if ``read_model`` is optimized as well. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_caching.cpp - :language: cpp - :fragment: [ov:caching:part2] - .. tab-item:: Python :sync: py @@ -110,6 +103,13 @@ With model caching enabled, the total load time is even shorter, if ``read_model :language: py :fragment: [ov:caching:part2] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_caching.cpp + :language: cpp + :fragment: [ov:caching:part2] + .. image:: _static/images/caching_times.svg @@ -121,13 +121,6 @@ To check in advance if a particular device supports model caching, your applicat .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_caching.cpp - :language: cpp - :fragment: [ov:caching:part3] - .. tab-item:: Python :sync: py @@ -135,6 +128,13 @@ To check in advance if a particular device supports model caching, your applicat :language: py :fragment: [ov:caching:part3] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_caching.cpp + :language: cpp + :fragment: [ov:caching:part3] + .. note:: diff --git a/docs/OV_Runtime_UG/ShapeInference.md b/docs/OV_Runtime_UG/ShapeInference.md index befb89d2558..b6e95b74ca8 100644 --- a/docs/OV_Runtime_UG/ShapeInference.md +++ b/docs/OV_Runtime_UG/ShapeInference.md @@ -34,13 +34,6 @@ to set a new batch size with the ``reshape`` method: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ShapeInference.cpp - :language: cpp - :fragment: picture_snippet - .. tab-item:: Python :sync: py @@ -48,6 +41,13 @@ to set a new batch size with the ``reshape`` method: :language: Python :fragment: picture_snippet + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ShapeInference.cpp + :language: cpp + :fragment: picture_snippet + The diagram below presents the results of using the method, where the size of model input is changed with an image input: @@ -63,13 +63,6 @@ When using the ``reshape`` method, you may take one of the approaches: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ShapeInference.cpp - :language: cpp - :fragment: spatial_reshape - .. tab-item:: Python :sync: py @@ -77,6 +70,13 @@ When using the ``reshape`` method, you may take one of the approaches: :language: python :fragment: simple_spatials_change + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ShapeInference.cpp + :language: cpp + :fragment: spatial_reshape + To do the opposite - to resize input image to match the input shapes of the model, use the :doc:`pre-processing API `. @@ -90,15 +90,6 @@ When using the ``reshape`` method, you may take one of the approaches: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - ``map, ov::PartialShape`` specifies input by passing actual input port: - - .. doxygensnippet:: docs/snippets/ShapeInference.cpp - :language: cpp - :fragment: [obj_to_shape] - .. tab-item:: Python :sync: py @@ -109,18 +100,18 @@ When using the ``reshape`` method, you may take one of the approaches: :language: python :fragment: [obj_to_shape] - .. tab-item:: Index - - .. tab-set:: - .. tab-item:: C++ :sync: cpp - ``map`` specifies input by its index: + ``map, ov::PartialShape`` specifies input by passing actual input port: .. doxygensnippet:: docs/snippets/ShapeInference.cpp :language: cpp - :fragment: [idx_to_shape] + :fragment: [obj_to_shape] + + .. tab-item:: Index + + .. tab-set:: .. tab-item:: Python :sync: py @@ -132,18 +123,18 @@ When using the ``reshape`` method, you may take one of the approaches: :language: python :fragment: [idx_to_shape] - .. tab-item:: Tensor Name - - .. tab-set:: - .. tab-item:: C++ :sync: cpp - ``map`` specifies input by its name: + ``map`` specifies input by its index: .. doxygensnippet:: docs/snippets/ShapeInference.cpp :language: cpp - :fragment: [name_to_shape] + :fragment: [idx_to_shape] + + .. tab-item:: Tensor Name + + .. tab-set:: .. tab-item:: Python :sync: py @@ -155,6 +146,15 @@ When using the ``reshape`` method, you may take one of the approaches: :language: python :fragment: [name_to_shape] + .. tab-item:: C++ + :sync: cpp + + ``map`` specifies input by its name: + + .. doxygensnippet:: docs/snippets/ShapeInference.cpp + :language: cpp + :fragment: [name_to_shape] + You can find the usage scenarios of the ``reshape`` method in :doc:`Hello Reshape SSD Samples `. @@ -173,13 +173,6 @@ To change the batch dimension of the model, :ref:`set the layout `). To explicitly specify which GPUs to use, set their priority when compiling with AUTO: - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. code-block:: sh - ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU.1,GPU.0", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT)); + ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU,CPU", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT)); + +If AUTO is used without specifying any device names, and if there are multiple GPUs in the system, CUMULATIVE_THROUGHPUT mode will use all of the GPUs by default. If the system has more than two GPU devices, AUTO will remove CPU from the device candidate list to keep the GPUs running at full capacity. A full list of system devices and their unique identifiers can be queried using ov::Core::get_available_devices (for more information, see :doc:`Query Device Properties `). To explicitly specify which GPUs to use, set their priority when compiling with AUTO: + +.. tab-set:: .. tab-item:: Python :sync: py @@ -332,6 +324,14 @@ If AUTO is used without specifying any device names, and if there are multiple G compiled_model = core.compile_model(model, "AUTO:GPU.1,GPU.0", {"PERFORMANCE_HINT" : {"CUMULATIVE_THROUGHPUT"}) + .. tab-item:: C++ + :sync: cpp + + .. code-block:: sh + + ov::CompiledModel compiled_model = core.compile_model(model, "AUTO:GPU.1,GPU.0", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT)); + + Code Examples -------------------- @@ -340,13 +340,6 @@ To enable performance hints for your application, use the following code: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/AUTO3.cpp - :language: cpp - :fragment: [part3] - .. tab-item:: Python :sync: py @@ -354,6 +347,13 @@ To enable performance hints for your application, use the following code: :language: python :fragment: [part3] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/AUTO3.cpp + :language: cpp + :fragment: [part3] + Disabling Auto-Batching for THROUGHPUT and CUMULATIVE_THROUGHPUT ---------------------------------------------------------------- @@ -368,13 +368,6 @@ The ``ov::hint::model_priority`` property enables you to control the priorities .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/AUTO4.cpp - :language: cpp - :fragment: [part4] - .. tab-item:: Python :sync: py @@ -382,6 +375,13 @@ The ``ov::hint::model_priority`` property enables you to control the priorities :language: python :fragment: [part4] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/AUTO4.cpp + :language: cpp + :fragment: [part4] + Checking Target Runtime Devices +++++++++++++++++++++++++++++++ @@ -391,13 +391,6 @@ To query the runtime target devices on which the inferences are being executed u .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/AUTO7.cpp - :language: cpp - :fragment: [part7] - .. tab-item:: Python :sync: py @@ -405,6 +398,12 @@ To query the runtime target devices on which the inferences are being executed u :language: python :fragment: [part7] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/AUTO7.cpp + :language: cpp + :fragment: [part7] Configuring Individual Devices and Creating the Auto-Device plugin on Top @@ -414,13 +413,6 @@ Although the methods described above are currently the preferred way to execute .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/AUTO5.cpp - :language: cpp - :fragment: [part5] - .. tab-item:: Python :sync: py @@ -428,6 +420,13 @@ Although the methods described above are currently the preferred way to execute :language: python :fragment: [part5] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/AUTO5.cpp + :language: cpp + :fragment: [part5] + .. _using-auto-with-openvino-samples-and-benchmark-app: diff --git a/docs/OV_Runtime_UG/automatic_batching.md b/docs/OV_Runtime_UG/automatic_batching.md index ee43cbcaf82..5b558502234 100644 --- a/docs/OV_Runtime_UG/automatic_batching.md +++ b/docs/OV_Runtime_UG/automatic_batching.md @@ -30,19 +30,19 @@ How Automatic Batching Works .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [compile_model] - .. tab-item:: Python :sync: py .. doxygensnippet:: docs/snippets/ov_auto_batching.py :language: Python :fragment: [compile_model] + + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [compile_model] To enable Auto-batching in the legacy apps not akin to the notion of performance hints, you need to use the **explicit** device notion, such as ``BATCH:GPU``. @@ -53,13 +53,6 @@ How Automatic Batching Works .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [compile_model_no_auto_batching] - .. tab-item:: Python :sync: py @@ -67,6 +60,13 @@ How Automatic Batching Works :language: Python :fragment: [compile_model_no_auto_batching] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [compile_model_no_auto_batching] + Configuring Automatic Batching ++++++++++++++++++++++++++++++ @@ -90,13 +90,6 @@ The support for Auto-batching is not limited to GPU. However, if a device does n This "automatic batch size selection" works on the presumption that the application queries ``ov::optimal_number_of_infer_requests`` to create the requests of the returned number and run them simultaneously: .. tab-set:: - - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [query_optimal_num_requests] .. tab-item:: Python :sync: py @@ -104,6 +97,13 @@ This "automatic batch size selection" works on the presumption that the applicat .. doxygensnippet:: docs/snippets/ov_auto_batching.py :language: Python :fragment: [query_optimal_num_requests] + + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [query_optimal_num_requests] .. _limiting-batch-size: @@ -118,13 +118,6 @@ For example, when the application processes only 4 video streams, there is no ne .. tab-set:: - - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [hint_num_requests] .. tab-item:: Python :sync: py @@ -132,6 +125,13 @@ For example, when the application processes only 4 video streams, there is no ne .. doxygensnippet:: docs/snippets/ov_auto_batching.py :language: Python :fragment: [hint_num_requests] + + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [hint_num_requests] For the *explicit* usage, you can limit the batch size by using ``BATCH:GPU(4)``, where 4 is the number of requests running in parallel. diff --git a/docs/OV_Runtime_UG/deployment/deployment-manager-tool.md b/docs/OV_Runtime_UG/deployment/deployment-manager-tool.md index 57c189e79cf..41695201355 100644 --- a/docs/OV_Runtime_UG/deployment/deployment-manager-tool.md +++ b/docs/OV_Runtime_UG/deployment/deployment-manager-tool.md @@ -47,14 +47,6 @@ Running Deployment Manager in Interactive Mode .. tab-set:: - .. tab-item:: Linux - :sync: linux - - .. code-block:: sh - - cd /tools/deployment_manager - ./deployment_manager.py - .. tab-item:: Windows :sync: windows @@ -62,6 +54,14 @@ Running Deployment Manager in Interactive Mode cd \tools\deployment_manager .\deployment_manager.py + + .. tab-item:: Linux + :sync: linux + + .. code-block:: sh + + cd /tools/deployment_manager + ./deployment_manager.py .. tab-item:: macOS :sync: macos @@ -109,14 +109,6 @@ Running Deployment Manager in Standard CLI Mode .. tab-set:: - .. tab-item:: Linux - :sync: linux - - .. code-block:: sh - - cd /tools/deployment_manager - ./deployment_manager.py <--targets> [--output_dir] [--archive_name] [--user_data] - .. tab-item:: Windows :sync: windows @@ -124,6 +116,14 @@ Running Deployment Manager in Standard CLI Mode cd \tools\deployment_manager .\deployment_manager.py <--targets> [--output_dir] [--archive_name] [--user_data] + + .. tab-item:: Linux + :sync: linux + + .. code-block:: sh + + cd /tools/deployment_manager + ./deployment_manager.py <--targets> [--output_dir] [--archive_name] [--user_data] .. tab-item:: macOS :sync: macos @@ -161,19 +161,19 @@ To deploy the OpenVINO Runtime components from the development machine to the ta .. tab-set:: - .. tab-item:: Linux - :sync: linux - - .. code-block:: sh - - tar xf openvino_deployment_package.tar.gz -C - .. tab-item:: Windows :sync: windows .. code-block:: bat Use the archiver of your choice to unzip the file. + + .. tab-item:: Linux + :sync: linux + + .. code-block:: sh + + tar xf openvino_deployment_package.tar.gz -C .. tab-item:: macOS :sync: macos @@ -202,14 +202,6 @@ To deploy the OpenVINO Runtime components from the development machine to the ta .. tab-set:: - .. tab-item:: Linux - :sync: linux - - .. code-block:: sh - - cd /openvino/ - source ./setupvars.sh - .. tab-item:: Windows :sync: windows @@ -217,6 +209,14 @@ To deploy the OpenVINO Runtime components from the development machine to the ta cd \openvino\ .\setupvars.bat + + .. tab-item:: Linux + :sync: linux + + .. code-block:: sh + + cd /openvino/ + source ./setupvars.sh .. tab-item:: macOS :sync: macos diff --git a/docs/OV_Runtime_UG/deployment/local-distribution.md b/docs/OV_Runtime_UG/deployment/local-distribution.md index 943e779ae22..77fec59fd06 100644 --- a/docs/OV_Runtime_UG/deployment/local-distribution.md +++ b/docs/OV_Runtime_UG/deployment/local-distribution.md @@ -51,6 +51,7 @@ As shown in the picture above, some plugin libraries may have OS-specific depend .. tab-set:: .. tab-item:: Windows + :sync: windows +--------------+-------------------------+-------------------------------------------------------+ | Device | Dependency | Location | @@ -67,7 +68,8 @@ As shown in the picture above, some plugin libraries may have OS-specific depend | Arm® CPU | — | — | +--------------+-------------------------+-------------------------------------------------------+ - .. tab-item:: Linux arm64 + .. tab-item:: Linux arm64 + :sync: linux-arm-64 +--------------+-------------------------+-------------------------------------------------------+ | Device | Dependency | Location | @@ -76,6 +78,7 @@ As shown in the picture above, some plugin libraries may have OS-specific depend +--------------+-------------------------+-------------------------------------------------------+ .. tab-item:: Linux x86_64 + :sync: linux-x86-64 +--------------+-------------------------+-------------------------------------------------------+ | Device | Dependency | Location | @@ -88,8 +91,8 @@ As shown in the picture above, some plugin libraries may have OS-specific depend | GNA | libgna.so | ``./runtime/lib/intel64/libgna.so.3`` | +--------------+-------------------------+-------------------------------------------------------+ - - .. tab-item:: macOS arm64 + .. tab-item:: macOS arm64 + :sync: macos-arm-64 +--------------+-------------------------+-------------------------------------------------------+ | Device | Dependency | Location | @@ -98,6 +101,7 @@ As shown in the picture above, some plugin libraries may have OS-specific depend +--------------+-------------------------+-------------------------------------------------------+ .. tab-item:: macOS x86_64 + :sync: macos-x86-64 +--------------+-------------------------+-------------------------------------------------------+ | Device | Dependency | Location | diff --git a/docs/OV_Runtime_UG/hetero_execution.md b/docs/OV_Runtime_UG/hetero_execution.md index 701ad34dd17..8310f30a276 100644 --- a/docs/OV_Runtime_UG/hetero_execution.md +++ b/docs/OV_Runtime_UG/hetero_execution.md @@ -48,13 +48,6 @@ Randomly selecting operations and setting affinities may lead to decrease in mod .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_hetero.cpp - :language: cpp - :fragment: [set_manual_affinities] - .. tab-item:: Python :sync: py @@ -62,6 +55,13 @@ Randomly selecting operations and setting affinities may lead to decrease in mod :language: Python :fragment: [set_manual_affinities] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_hetero.cpp + :language: cpp + :fragment: [set_manual_affinities] + The Automatic Mode -------------------- @@ -74,13 +74,6 @@ It does not take into account device peculiarities such as the inability to infe .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_hetero.cpp - :language: cpp - :fragment: [compile_model] - .. tab-item:: Python :sync: py @@ -88,6 +81,13 @@ It does not take into account device peculiarities such as the inability to infe :language: Python :fragment: [compile_model] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_hetero.cpp + :language: cpp + :fragment: [compile_model] + Using Manual and Automatic Modes in Combination ----------------------------------------------- @@ -97,13 +97,6 @@ In some cases you may need to consider manually adjusting affinities which were .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_hetero.cpp - :language: cpp - :fragment: [fix_automatic_affinities] - .. tab-item:: Python :sync: py @@ -111,6 +104,13 @@ In some cases you may need to consider manually adjusting affinities which were :language: Python :fragment: [fix_automatic_affinities] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_hetero.cpp + :language: cpp + :fragment: [fix_automatic_affinities] + Importantly, the automatic mode will not work if any operation in a model has its ``"affinity"`` already initialized. @@ -126,13 +126,6 @@ If you want different devices in Hetero execution to have different device-speci .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_hetero.cpp - :language: cpp - :fragment: [configure_fallback_devices] - .. tab-item:: Python :sync: py @@ -140,6 +133,13 @@ If you want different devices in Hetero execution to have different device-speci :language: Python :fragment: [configure_fallback_devices] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_hetero.cpp + :language: cpp + :fragment: [configure_fallback_devices] + In the example above, the ``GPU`` device is configured to enable profiling data and uses the default execution precision, while ``CPU`` has the configuration property to perform inference in ``fp32``. diff --git a/docs/OV_Runtime_UG/integrate_with_your_application.md b/docs/OV_Runtime_UG/integrate_with_your_application.md index c102709bed2..9e006270b8b 100644 --- a/docs/OV_Runtime_UG/integrate_with_your_application.md +++ b/docs/OV_Runtime_UG/integrate_with_your_application.md @@ -33,13 +33,6 @@ Include next files to work with OpenVINO™ Runtime: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [include] - .. tab-item:: Python :sync: py @@ -47,6 +40,13 @@ Include next files to work with OpenVINO™ Runtime: :language: python :fragment: [import] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [include] + .. tab-item:: C :sync: c @@ -59,13 +59,6 @@ Use the following code to create OpenVINO™ Core to manage available devices an .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part1] - .. tab-item:: Python :sync: py @@ -73,6 +66,13 @@ Use the following code to create OpenVINO™ Core to manage available devices an :language: python :fragment: [part1] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part1] + .. tab-item:: C :sync: c @@ -90,53 +90,6 @@ Compile the model for a specific device using ``ov::Core::compile_model()``: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. tab-set:: - - .. tab-item:: IR - :sync: ir - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_1] - - .. tab-item:: ONNX - :sync: onnx - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_2] - - .. tab-item:: PaddlePaddle - :sync: pdpd - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_3] - - .. tab-item:: TensorFlow - :sync: tf - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_4] - - .. tab-item:: TensorFlow Lite - :sync: tf_lite - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_5] - - .. tab-item:: ov::Model - :sync: ovmodel - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part2_6] - .. tab-item:: Python :sync: py @@ -157,33 +110,80 @@ Compile the model for a specific device using ``ov::Core::compile_model()``: :fragment: [part2_2] .. tab-item:: PaddlePaddle - :sync: pdpd + :sync: paddlepaddle .. doxygensnippet:: docs/snippets/src/main.py :language: python :fragment: [part2_3] .. tab-item:: TensorFlow - :sync: tf + :sync: tensorflow .. doxygensnippet:: docs/snippets/src/main.py :language: python :fragment: [part2_4] .. tab-item:: TensorFlow Lite - :sync: tf_lite + :sync: tensorflow_lite .. doxygensnippet:: docs/snippets/src/main.py :language: python :fragment: [part2_5] .. tab-item:: ov::Model - :sync: ovmodel + :sync: openvinomodel .. doxygensnippet:: docs/snippets/src/main.py :language: python :fragment: [part2_6] + .. tab-item:: C++ + :sync: cpp + + .. tab-set:: + + .. tab-item:: IR + :sync: ir + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_1] + + .. tab-item:: ONNX + :sync: onnx + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_2] + + .. tab-item:: PaddlePaddle + :sync: paddlepaddle + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_3] + + .. tab-item:: TensorFlow + :sync: tensorflow + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_4] + + .. tab-item:: TensorFlow Lite + :sync: tensorflow_lite + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_5] + + .. tab-item:: ov::Model + :sync: openvinomodel + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part2_6] + .. tab-item:: C :sync: c @@ -204,28 +204,28 @@ Compile the model for a specific device using ``ov::Core::compile_model()``: :fragment: [part2_2] .. tab-item:: PaddlePaddle - :sync: pdpd + :sync: paddlepaddle .. doxygensnippet:: docs/snippets/src/main.c :language: cpp :fragment: [part2_3] .. tab-item:: TensorFlow - :sync: tf + :sync: tensorflow .. doxygensnippet:: docs/snippets/src/main.c :language: cpp :fragment: [part2_4] .. tab-item:: TensorFlow Lite - :sync: tf_lite + :sync: tensorflow_lite .. doxygensnippet:: docs/snippets/src/main.c :language: cpp :fragment: [part2_5] .. tab-item:: ov::Model - :sync: ovmodel + :sync: openvinomodel .. doxygensnippet:: docs/snippets/src/main.c :language: cpp @@ -248,13 +248,6 @@ Create an infer request using the following code (see .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part3] - .. tab-item:: Python :sync: py @@ -262,6 +255,13 @@ Create an infer request using the following code (see :language: python :fragment: [part3] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part3] + .. tab-item:: C :sync: c @@ -277,13 +277,6 @@ You can use external memory to create ``ov::Tensor`` and use the ``ov::InferRequ .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part4] - .. tab-item:: Python :sync: py @@ -291,6 +284,13 @@ You can use external memory to create ``ov::Tensor`` and use the ``ov::InferRequ :language: python :fragment: [part4] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part4] + .. tab-item:: C :sync: c @@ -306,13 +306,6 @@ OpenVINO™ Runtime supports inference in either synchronous or asynchronous mod .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part5] - .. tab-item:: Python :sync: py @@ -320,6 +313,13 @@ OpenVINO™ Runtime supports inference in either synchronous or asynchronous mod :language: python :fragment: [part5] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part5] + .. tab-item:: C :sync: c @@ -338,13 +338,6 @@ Go over the output tensors and process the inference results. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/src/main.cpp - :language: cpp - :fragment: [part6] - .. tab-item:: Python :sync: py @@ -352,6 +345,13 @@ Go over the output tensors and process the inference results. :language: python :fragment: [part6] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/src/main.cpp + :language: cpp + :fragment: [part6] + .. tab-item:: C :sync: c diff --git a/docs/OV_Runtime_UG/layout_overview.md b/docs/OV_Runtime_UG/layout_overview.md index 40a8eabba8a..567b7da8305 100644 --- a/docs/OV_Runtime_UG/layout_overview.md +++ b/docs/OV_Runtime_UG/layout_overview.md @@ -33,13 +33,6 @@ The easiest way is to fully specify each dimension with one alphabet letter. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_layout.cpp - :language: cpp - :fragment: ov:layout:simple - .. tab-item:: Python :sync: py @@ -47,6 +40,13 @@ The easiest way is to fully specify each dimension with one alphabet letter. :language: python :fragment: ov:layout:simple + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_layout.cpp + :language: cpp + :fragment: ov:layout:simple + This assigns ``N`` to the first dimension, ``C`` to the second, ``H`` to the third, and ``W`` to the fourth. @@ -58,13 +58,6 @@ The advanced syntax allows assigning a word to a dimension. To do this, wrap a l .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_layout.cpp - :language: cpp - :fragment: ov:layout:complex - .. tab-item:: Python :sync: py @@ -72,6 +65,13 @@ The advanced syntax allows assigning a word to a dimension. To do this, wrap a l :language: python :fragment: ov:layout:complex + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_layout.cpp + :language: cpp + :fragment: ov:layout:complex + Partially Defined Layout ++++++++++++++++++++++++ @@ -81,13 +81,6 @@ If a certain dimension is not important, its name can be set to ``?``. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_layout.cpp - :language: cpp - :fragment: ov:layout:partially_defined - .. tab-item:: Python :sync: py @@ -95,6 +88,13 @@ If a certain dimension is not important, its name can be set to ``?``. :language: python :fragment: ov:layout:partially_defined + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_layout.cpp + :language: cpp + :fragment: ov:layout:partially_defined + Dynamic Layout ++++++++++++++++++++ @@ -104,13 +104,6 @@ If several dimensions are not important, an ellipsis ``...`` can be used to spec .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_layout.cpp - :language: cpp - :fragment: ov:layout:dynamic - .. tab-item:: Python :sync: py @@ -118,6 +111,13 @@ If several dimensions are not important, an ellipsis ``...`` can be used to spec :language: python :fragment: ov:layout:dynamic + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_layout.cpp + :language: cpp + :fragment: ov:layout:dynamic + Predefined Names ++++++++++++++++++++ @@ -135,13 +135,6 @@ These names are used in :doc:`PreProcessing API ` section. @@ -59,13 +59,6 @@ OpenVINO™ Runtime provides two types for shape representation: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp - :language: cpp - :fragment: [ov:partial_shape] - .. tab-item:: Python :sync: py @@ -73,6 +66,13 @@ OpenVINO™ Runtime provides two types for shape representation: :language: cpp :fragment: [ov:partial_shape] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp + :language: cpp + :fragment: [ov:partial_shape] + However, in most cases, before getting static shape using the ``get_shape()`` method, you need to check if that shape is static. @@ -104,13 +104,6 @@ To build an ``:ref:`ov::Model ``` instance from ``opset .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp - :language: cpp - :fragment: [ov:include] - .. tab-item:: Python :sync: py @@ -118,17 +111,17 @@ To build an ``:ref:`ov::Model ``` instance from ``opset :language: cpp :fragment: [import] - -The following code demonstrates how to create a simple model: - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp :language: cpp - :fragment: [ov:create_simple_model] + :fragment: [ov:include] + + +The following code demonstrates how to create a simple model: + +.. tab-set:: .. tab-item:: Python :sync: py @@ -137,18 +130,18 @@ The following code demonstrates how to create a simple model: :language: cpp :fragment: [ov:create_simple_model] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp + :language: cpp + :fragment: [ov:create_simple_model] + The following code creates a model with several outputs: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp - :language: cpp - :fragment: [ov:create_advanced_model] - .. tab-item:: Python :sync: py @@ -156,6 +149,13 @@ The following code creates a model with several outputs: :language: cpp :fragment: [ov:create_advanced_model] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp + :language: cpp + :fragment: [ov:create_advanced_model] + Model Debugging Capabilities ########################################### @@ -168,13 +168,6 @@ OpenVINO™ provides several debug capabilities: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp - :language: cpp - :fragment: [ov:visualize] - .. tab-item:: Python :sync: py @@ -182,6 +175,13 @@ OpenVINO™ provides several debug capabilities: :language: cpp :fragment: [ov:visualize] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp + :language: cpp + :fragment: [ov:visualize] + .. code-block:: sh @@ -204,13 +204,6 @@ OpenVINO™ provides several debug capabilities: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp - :language: cpp - :fragment: [ov:serialize] - .. tab-item:: Python :sync: py @@ -218,6 +211,13 @@ OpenVINO™ provides several debug capabilities: :language: cpp :fragment: [ov:serialize] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_model_snippets.cpp + :language: cpp + :fragment: [ov:serialize] + Additional Resources ######################## diff --git a/docs/OV_Runtime_UG/multi_device.md b/docs/OV_Runtime_UG/multi_device.md index 4cfb4078915..aaa194e9092 100644 --- a/docs/OV_Runtime_UG/multi_device.md +++ b/docs/OV_Runtime_UG/multi_device.md @@ -53,13 +53,6 @@ The following commands are accepted by the API: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/MULTI0.cpp - :language: cpp - :fragment: [part0] - .. tab-item:: Python :sync: py @@ -67,6 +60,13 @@ The following commands are accepted by the API: :language: python :fragment: [MULTI_0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/MULTI0.cpp + :language: cpp + :fragment: [part0] + To check what devices are present in the system, you can use the Device API. For information on how to do it, check :doc:`Query device properties and configuration `. @@ -78,13 +78,6 @@ As mentioned previously, executing inference with MULTI may be set up by configu .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/MULTI4.cpp - :language: cpp - :fragment: [part4] - .. tab-item:: Python :sync: py @@ -92,6 +85,13 @@ As mentioned previously, executing inference with MULTI may be set up by configu :language: python :fragment: [MULTI_4] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/MULTI4.cpp + :language: cpp + :fragment: [part4] + Alternatively, you can combine all the individual device settings into a single config file and load it for MULTI to parse. See the code example in the next section. diff --git a/docs/OV_Runtime_UG/ov_dynamic_shapes.md b/docs/OV_Runtime_UG/ov_dynamic_shapes.md index e77d5130360..782029223db 100644 --- a/docs/OV_Runtime_UG/ov_dynamic_shapes.md +++ b/docs/OV_Runtime_UG/ov_dynamic_shapes.md @@ -66,13 +66,6 @@ The examples below show how to set dynamic dimensions with a model that has a st .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: [ov_dynamic_shapes:reshape_undefined] - .. tab-item:: Python :sync: py @@ -82,6 +75,13 @@ The examples below show how to set dynamic dimensions with a model that has a st With Python, you may also pass all dimensions as a string and use ``?`` for the dynamic dimensions (e.g. ``model.reshape(“1, 3, ?, ?”)``). + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: [ov_dynamic_shapes:reshape_undefined] + .. tab-item:: C :sync: c @@ -96,13 +96,6 @@ The examples above assume that the model has a single input layer. To change mod .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:reshape_multiple_inputs - .. tab-item:: Python :sync: py @@ -110,6 +103,13 @@ The examples above assume that the model has a single input layer. To change mod :language: python :fragment: reshape_multiple_inputs + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:reshape_multiple_inputs + For more examples of how to change multiple input layers, see :doc:`Changing Input Shapes `. @@ -122,13 +122,6 @@ To check if a model already has dynamic dimensions, first load it with the ``rea .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:check_inputs - .. tab-item:: Python :sync: py @@ -136,6 +129,13 @@ To check if a model already has dynamic dimensions, first load it with the ``rea :language: python :fragment: check_inputs + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:check_inputs + If the input model already has dynamic dimensions, that will not change during inference. If the inputs will not be used dynamically, it is recommended to set them to static values using the ``reshape`` method to save application memory and potentially improve inference speed. The OpenVINO API supports any combination of static and dynamic dimensions. @@ -147,16 +147,7 @@ Dimension Bounds The lower and/or upper bounds of a dynamic dimension can also be specified. They define a range of allowed values for the dimension. Dimension bounds can be set by passing the lower and upper bounds into the ``reshape`` method using the options shown below. .. tab-set:: - - .. tab-item:: C++ - :sync: cpp - - The dimension bounds can be coded as arguments for ``ov::Dimension``, as shown in these examples: - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:reshape_bounds - + .. tab-item:: Python :sync: py @@ -172,6 +163,15 @@ The lower and/or upper bounds of a dynamic dimension can also be specified. They :language: python :fragment: reshape_bounds + .. tab-item:: C++ + :sync: cpp + + The dimension bounds can be coded as arguments for ``ov::Dimension``, as shown in these examples: + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:reshape_bounds + .. tab-item:: C :sync: c @@ -203,19 +203,19 @@ The sample below shows how a model can accept different input shapes. In the fir .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:set_input_tensor - .. tab-item:: Python :sync: py .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py :language: python :fragment: set_input_tensor + + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:set_input_tensor .. tab-item:: C :sync: c @@ -238,13 +238,6 @@ To determine if the output has dynamic dimensions, the ``partial_shape`` propert .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:print_dynamic - .. tab-item:: Python :sync: py @@ -252,6 +245,13 @@ To determine if the output has dynamic dimensions, the ``partial_shape`` propert :language: python :fragment: print_dynamic + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:print_dynamic + .. tab-item:: C :sync: c @@ -267,13 +267,6 @@ Output layers can also be checked for dynamic dimensions using the ``partial_sha .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp - :language: cpp - :fragment: ov_dynamic_shapes:detect_dynamic - .. tab-item:: Python :sync: py @@ -281,6 +274,13 @@ Output layers can also be checked for dynamic dimensions using the ``partial_sha :language: python :fragment: detect_dynamic + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp + :language: cpp + :fragment: ov_dynamic_shapes:detect_dynamic + .. tab-item:: C :sync: c diff --git a/docs/OV_Runtime_UG/ov_infer_request.md b/docs/OV_Runtime_UG/ov_infer_request.md index ad5723e72c4..6779a3bbcee 100644 --- a/docs/OV_Runtime_UG/ov_infer_request.md +++ b/docs/OV_Runtime_UG/ov_infer_request.md @@ -20,13 +20,6 @@ The ``ov::InferRequest`` can be created from the ``ov::CompiledModel``: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: create_infer_request - .. tab-item:: Python :sync: py @@ -34,6 +27,13 @@ The ``ov::InferRequest`` can be created from the ``ov::CompiledModel``: :language: python :fragment: create_infer_request + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: create_infer_request + Run Inference #################### @@ -48,13 +48,6 @@ You can use ``ov::InferRequest::infer``, which blocks the application execution, .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: sync_infer - .. tab-item:: Python :sync: py @@ -62,6 +55,13 @@ You can use ``ov::InferRequest::infer``, which blocks the application execution, :language: python :fragment: sync_infer + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: sync_infer + Asynchronous Mode ++++++++++++++++++++ @@ -71,13 +71,6 @@ The asynchronous mode can improve application's overall frame-rate, by making it .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: async_infer - .. tab-item:: Python :sync: py @@ -85,6 +78,13 @@ The asynchronous mode can improve application's overall frame-rate, by making it :language: python :fragment: async_infer + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: async_infer + Asynchronous mode supports two ways the application waits for inference results: @@ -92,13 +92,6 @@ Asynchronous mode supports two ways the application waits for inference results: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: wait_for - .. tab-item:: Python :sync: py @@ -106,11 +99,25 @@ Asynchronous mode supports two ways the application waits for inference results: :language: python :fragment: wait_for + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: wait_for + * ``ov::InferRequest::wait`` - waits until inference result becomes available .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_infer_request.py + :language: python + :fragment: wait + .. tab-item:: C++ :sync: cpp @@ -118,12 +125,6 @@ Asynchronous mode supports two ways the application waits for inference results: :language: cpp :fragment: wait - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/ov_infer_request.py - :language: python - :fragment: wait Both methods are thread-safe. @@ -133,13 +134,6 @@ When you are running several inference requests in parallel, a device can proces .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: set_callback - .. tab-item:: Python :sync: py @@ -147,6 +141,13 @@ When you are running several inference requests in parallel, a device can proces :language: python :fragment: set_callback + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: set_callback + .. note:: @@ -160,13 +161,6 @@ You can use the ``ov::InferRequest::cancel`` method if you want to abort executi .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: cancel - .. tab-item:: Python :sync: py @@ -174,6 +168,13 @@ You can use the ``ov::InferRequest::cancel`` method if you want to abort executi :language: python :fragment: cancel + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: cancel + .. _in_out_tensors: @@ -187,13 +188,6 @@ Working with Input and Output tensors .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: get_set_one_tensor - .. tab-item:: Python :sync: py @@ -201,6 +195,12 @@ Working with Input and Output tensors :language: python :fragment: get_set_one_tensor + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: get_set_one_tensor * ``ov::InferRequest::get_input_tensor``, ``ov::InferRequest::set_input_tensor``, ``ov::InferRequest::get_output_tensor``, ``ov::InferRequest::set_output_tensor`` methods with argument can be used to get or set input/output tensor by input/output index: @@ -208,13 +208,6 @@ Working with Input and Output tensors .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: get_set_index_tensor - .. tab-item:: Python :sync: py @@ -222,18 +215,18 @@ Working with Input and Output tensors :language: python :fragment: get_set_index_tensor - -* ``ov::InferRequest::get_tensor``, ``ov::InferRequest::set_tensor`` methods can be used to get or set input/output tensor by tensor name: - - - .. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_infer_request.cpp :language: cpp - :fragment: get_set_tensor + :fragment: get_set_index_tensor + + +* ``ov::InferRequest::get_tensor``, ``ov::InferRequest::set_tensor`` methods can be used to get or set input/output tensor by tensor name: + + + .. tab-set:: .. tab-item:: Python :sync: py @@ -242,18 +235,18 @@ Working with Input and Output tensors :language: python :fragment: get_set_tensor - -* ``ov::InferRequest::get_tensor``, ``ov::InferRequest::set_tensor`` methods can be used to get or set input/output tensor by port: - - - .. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_infer_request.cpp :language: cpp - :fragment: get_set_tensor_by_port + :fragment: get_set_tensor + + +* ``ov::InferRequest::get_tensor``, ``ov::InferRequest::set_tensor`` methods can be used to get or set input/output tensor by port: + + + .. tab-set:: .. tab-item:: Python :sync: py @@ -262,6 +255,13 @@ Working with Input and Output tensors :language: python :fragment: get_set_tensor_by_port + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: get_set_tensor_by_port + Examples of Infer Request Usages ################################ @@ -277,13 +277,6 @@ In this case, you can get the output tensor from the first request, using ``ov:: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: cascade_models - .. tab-item:: Python :sync: py @@ -291,6 +284,13 @@ In this case, you can get the output tensor from the first request, using ``ov:: :language: python :fragment: cascade_models + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: cascade_models + Using of ROI Tensors ++++++++++++++++++++ @@ -300,13 +300,6 @@ It is possible to re-use shared input in several models. You do not need to allo .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: roi_tensor - .. tab-item:: Python :sync: py @@ -315,6 +308,13 @@ It is possible to re-use shared input in several models. You do not need to allo :fragment: roi_tensor + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: roi_tensor + Using Remote Tensors ++++++++++++++++++++ @@ -323,13 +323,6 @@ By using ``ov::RemoteContext`` you can create a remote tensor to work with remot .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_infer_request.cpp - :language: cpp - :fragment: remote_tensor - .. tab-item:: Python :sync: py @@ -337,5 +330,12 @@ By using ``ov::RemoteContext`` you can create a remote tensor to work with remot :language: python :fragment: remote_tensor + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_infer_request.cpp + :language: cpp + :fragment: remote_tensor + @endsphinxdirective diff --git a/docs/OV_Runtime_UG/performance_hints.md b/docs/OV_Runtime_UG/performance_hints.md index 71077748ab5..4d9e5509248 100644 --- a/docs/OV_Runtime_UG/performance_hints.md +++ b/docs/OV_Runtime_UG/performance_hints.md @@ -57,13 +57,6 @@ In the example code snippet below, ``ov::hint::PerformanceMode::THROUGHPUT`` is .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [compile_model] - .. tab-item:: Python :sync: py @@ -71,6 +64,13 @@ In the example code snippet below, ``ov::hint::PerformanceMode::THROUGHPUT`` is :language: python :fragment: [compile_model] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [compile_model] + Additional (Optional) Hints from the App ######################################## @@ -80,13 +80,6 @@ As mentioned earlier, this will limit the batch size for the GPU and the number .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [hint_num_requests] - .. tab-item:: Python :sync: py @@ -94,6 +87,13 @@ As mentioned earlier, this will limit the batch size for the GPU and the number :language: python :fragment: [hint_num_requests] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [hint_num_requests] + Optimal Number of Inference Requests #################################### @@ -102,13 +102,6 @@ The hints are used on the presumption that the application queries ``ov::optimal .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [query_optimal_num_requests] - .. tab-item:: Python :sync: py @@ -116,6 +109,13 @@ The hints are used on the presumption that the application queries ``ov::optimal :language: python :fragment: [query_optimal_num_requests] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [query_optimal_num_requests] + While an application is free to create more requests if needed (for example to support asynchronous inputs population) **it is very important to at least run the** ``ov::optimal_number_of_infer_requests`` **of the inference requests in parallel**. It is recommended for efficiency, or device utilization, reasons. @@ -143,13 +143,6 @@ For example, use ``ov::hint::PerformanceMode::THROUGHPUT`` to prepare a general .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp - :language: cpp - :fragment: [hint_plus_low_level] - .. tab-item:: Python :sync: py @@ -157,6 +150,13 @@ For example, use ``ov::hint::PerformanceMode::THROUGHPUT`` to prepare a general :language: python :fragment: [hint_plus_low_level] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp + :language: cpp + :fragment: [hint_plus_low_level] + Testing Performance of the Hints with the Benchmark_App ####################################################### diff --git a/docs/OV_Runtime_UG/precision_control.md b/docs/OV_Runtime_UG/precision_control.md index 20e09ba63ad..12126959b2c 100644 --- a/docs/OV_Runtime_UG/precision_control.md +++ b/docs/OV_Runtime_UG/precision_control.md @@ -30,13 +30,6 @@ Code examples: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/cpu/ov_execution_mode.cpp - :language: cpp - :fragment: [ov:execution_mode:part0] - .. tab-item:: Python :sync: py @@ -44,6 +37,14 @@ Code examples: :language: python :fragment: [ov:execution_mode:part0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/cpu/ov_execution_mode.cpp + :language: cpp + :fragment: [ov:execution_mode:part0] + + Inference Precision ################### diff --git a/docs/OV_Runtime_UG/preprocessing_details.md b/docs/OV_Runtime_UG/preprocessing_details.md index 414eaf972d1..e3192f3b8cc 100644 --- a/docs/OV_Runtime_UG/preprocessing_details.md +++ b/docs/OV_Runtime_UG/preprocessing_details.md @@ -21,13 +21,6 @@ If the model has only one input, then simple `ov::preprocess::PrePostProcessor:: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp - :language: cpp - :fragment: ov:preprocess:input_1 - .. tab-item:: Python :sync: py @@ -35,18 +28,18 @@ If the model has only one input, then simple `ov::preprocess::PrePostProcessor:: :language: python :fragment: ov:preprocess:input_1 - -In general, when a model has multiple inputs/outputs, each one can be addressed by a tensor name. - - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp :language: cpp - :fragment: ov:preprocess:input_name + :fragment: ov:preprocess:input_1 + + +In general, when a model has multiple inputs/outputs, each one can be addressed by a tensor name. + + +.. tab-set:: .. tab-item:: Python :sync: py @@ -55,17 +48,17 @@ In general, when a model has multiple inputs/outputs, each one can be addressed :language: python :fragment: ov:preprocess:input_name - -Or by it's index. - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp :language: cpp - :fragment: ov:preprocess:input_index + :fragment: ov:preprocess:input_name + + +Or by it's index. + +.. tab-set:: .. tab-item:: Python :sync: py @@ -74,6 +67,13 @@ Or by it's index. :language: python :fragment: ov:preprocess:input_index + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp + :language: cpp + :fragment: ov:preprocess:input_index + C++ references: @@ -96,13 +96,6 @@ Typical data normalization includes 2 operations for each data item: subtract me .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp - :language: cpp - :fragment: ov:preprocess:mean_scale - .. tab-item:: Python :sync: py @@ -110,18 +103,18 @@ Typical data normalization includes 2 operations for each data item: subtract me :language: python :fragment: ov:preprocess:mean_scale - -In Computer Vision area normalization is usually done separately for R, G, B values. To do this, :doc:`layout with 'C' dimension ` shall be defined. Example: - - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp :language: cpp - :fragment: ov:preprocess:mean_scale_array + :fragment: ov:preprocess:mean_scale + + +In Computer Vision area normalization is usually done separately for R, G, B values. To do this, :doc:`layout with 'C' dimension ` shall be defined. Example: + + +.. tab-set:: .. tab-item:: Python :sync: py @@ -130,6 +123,13 @@ In Computer Vision area normalization is usually done separately for R, G, B val :language: python :fragment: ov:preprocess:mean_scale_array + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp + :language: cpp + :fragment: ov:preprocess:mean_scale_array + C++ references: @@ -147,6 +147,13 @@ To integrate precision conversion into an execution graph as a pre-processing st .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_preprocessing.py + :language: python + :fragment: ov:preprocess:convert_element_type + .. tab-item:: C++ :sync: cpp @@ -154,12 +161,7 @@ To integrate precision conversion into an execution graph as a pre-processing st :language: cpp :fragment: ov:preprocess:convert_element_type - .. tab-item:: Python - :sync: py - .. doxygensnippet:: docs/snippets/ov_preprocessing.py - :language: python - :fragment: ov:preprocess:convert_element_type C++ references: @@ -178,13 +180,6 @@ Conversion can be done implicitly, using the :doc:`layout ` in the code: - - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/ov_preprocessing.cpp :language: cpp - :fragment: ov:preprocess:convert_layout_2 + :fragment: ov:preprocess:convert_layout + + +For a manual transpose of axes without the use of a :doc:`layout ` in the code: + + +.. tab-set:: .. tab-item:: Python :sync: py @@ -212,6 +207,13 @@ For a manual transpose of axes without the use of a :doc:`layout ` article. @@ -256,13 +256,6 @@ with the static input shape to get the best performance. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/cpu/dynamic_shape.cpp - :language: cpp - :fragment: [static_shape] - .. tab-item:: Python :sync: py @@ -270,6 +263,13 @@ with the static input shape to get the best performance. :language: py :fragment: [static_shape] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/cpu/dynamic_shape.cpp + :language: cpp + :fragment: [static_shape] + For more details, see the :doc:`dynamic shapes guide `. @@ -388,6 +388,13 @@ User can use the following properties to limit available CPU resource for model .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/cpu/multi_threading.py + :language: python + :fragment: [ov:intel_cpu:multi_threading:part0] + .. tab-item:: C++ :sync: cpp @@ -395,12 +402,6 @@ User can use the following properties to limit available CPU resource for model :language: cpp :fragment: [ov:intel_cpu:multi_threading:part0] - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/cpu/multi_threading.py - :language: python - :fragment: [ov:intel_cpu:multi_threading:part0] .. note:: @@ -410,6 +411,13 @@ By default, OpenVINO Runtime will enable CPU threads pinning for better performa .. tab-set:: + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/cpu/multi_threading.py + :language: python + :fragment: [ov:intel_cpu:multi_threading:part1] + .. tab-item:: C++ :sync: cpp @@ -417,12 +425,6 @@ By default, OpenVINO Runtime will enable CPU threads pinning for better performa :language: cpp :fragment: [ov:intel_cpu:multi_threading:part1] - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/cpu/multi_threading.py - :language: python - :fragment: [ov:intel_cpu:multi_threading:part1] user can check the :doc:`optimization guide ` for details on multi-stream execution @@ -460,13 +462,6 @@ To enable denormals optimization in the application, the ``denormals_optimizatio .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_denormals.cpp - :language: cpp - :fragment: [ov:intel_cpu:denormals_optimization:part0] - .. tab-item:: Python :sync: py @@ -474,6 +469,13 @@ To enable denormals optimization in the application, the ``denormals_optimizatio :language: python :fragment: [ov:intel_cpu:denormals_optimization:part0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_denormals.cpp + :language: cpp + :fragment: [ov:intel_cpu:denormals_optimization:part0] + Sparse weights decompression (Intel® x86-64) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ @@ -504,13 +506,6 @@ Code examples of how to use ``sparse_weights_decompression_rate``: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/cpu/ov_sparse_weights_decompression.cpp - :language: cpp - :fragment: [ov:intel_cpu:sparse_weights_decompression:part0] - .. tab-item:: Python :sync: py @@ -518,6 +513,13 @@ Code examples of how to use ``sparse_weights_decompression_rate``: :language: python :fragment: [ov:intel_cpu:sparse_weights_decompression:part0] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/cpu/ov_sparse_weights_decompression.cpp + :language: cpp + :fragment: [ov:intel_cpu:sparse_weights_decompression:part0] + .. note:: diff --git a/docs/OV_Runtime_UG/supported_plugins/GNA.md b/docs/OV_Runtime_UG/supported_plugins/GNA.md index c44f819839f..442ee6f09db 100644 --- a/docs/OV_Runtime_UG/supported_plugins/GNA.md +++ b/docs/OV_Runtime_UG/supported_plugins/GNA.md @@ -85,17 +85,6 @@ Therefore, there is no need for explicitly switching between GNA and CPU. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gna/configure.cpp - :language: cpp - :fragment: [include] - - .. doxygensnippet:: docs/snippets/gna/configure.cpp - :language: cpp - :fragment: [ov_gna_exec_mode_hw_with_sw_fback] - .. tab-item:: Python :sync: py @@ -107,6 +96,17 @@ Therefore, there is no need for explicitly switching between GNA and CPU. :language: py :fragment: [ov_gna_exec_mode_hw_with_sw_fback] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gna/configure.cpp + :language: cpp + :fragment: [include] + + .. doxygensnippet:: docs/snippets/gna/configure.cpp + :language: cpp + :fragment: [ov_gna_exec_mode_hw_with_sw_fback] + .. note:: @@ -163,13 +163,6 @@ To export a model for a specific version of GNA HW, use the ``ov::intel_gna::com .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gna/import_export.cpp - :language: cpp - :fragment: [ov_gna_export] - .. tab-item:: Python :sync: py @@ -177,19 +170,19 @@ To export a model for a specific version of GNA HW, use the ``ov::intel_gna::com :language: py :fragment: [ov_gna_export] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gna/import_export.cpp + :language: cpp + :fragment: [ov_gna_export] + Import model: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gna/import_export.cpp - :language: cpp - :fragment: [ov_gna_import] - .. tab-item:: Python :sync: py @@ -197,6 +190,13 @@ Import model: :language: py :fragment: [ov_gna_import] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gna/import_export.cpp + :language: cpp + :fragment: [ov_gna_import] + To compile a model, use either :ref:`compile Tool ` or :doc:`Speech C++ Sample `. @@ -220,16 +220,16 @@ With the following methods, you can collect profiling information with various p .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - ``ov::InferRequest::get_profiling_info`` - .. tab-item:: Python :sync: py ``openvino.runtime.InferRequest.get_profiling_info`` + .. tab-item:: C++ + :sync: cpp + + ``ov::InferRequest::get_profiling_info`` + The current GNA implementation calculates counters for the whole utterance scoring and does not provide per-layer information. The API enables you to retrieve counter units in cycles. You can convert cycles to seconds as follows: @@ -368,17 +368,6 @@ To set the layout of model inputs in runtime, use the :doc:`Optimize Preprocessi .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gna/set_batch.cpp - :language: cpp - :fragment: [include] - - .. doxygensnippet:: docs/snippets/gna/set_batch.cpp - :language: cpp - :fragment: [ov_gna_set_nc_layout] - .. tab-item:: Python :sync: py @@ -390,17 +379,21 @@ To set the layout of model inputs in runtime, use the :doc:`Optimize Preprocessi :language: py :fragment: [ov_gna_set_nc_layout] - -then set batch size: - -.. tab-set:: - .. tab-item:: C++ :sync: cpp .. doxygensnippet:: docs/snippets/gna/set_batch.cpp :language: cpp - :fragment: [ov_gna_set_batch_size] + :fragment: [include] + + .. doxygensnippet:: docs/snippets/gna/set_batch.cpp + :language: cpp + :fragment: [ov_gna_set_nc_layout] + + +then set batch size: + +.. tab-set:: .. tab-item:: Python :sync: py @@ -409,6 +402,13 @@ then set batch size: :language: py :fragment: [ov_gna_set_batch_size] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gna/set_batch.cpp + :language: cpp + :fragment: [ov_gna_set_batch_size] + Increasing batch size only improves efficiency of ``MatMul`` layers. diff --git a/docs/OV_Runtime_UG/supported_plugins/GPU.md b/docs/OV_Runtime_UG/supported_plugins/GPU.md index cd51350cc7d..eafe3aad406 100644 --- a/docs/OV_Runtime_UG/supported_plugins/GPU.md +++ b/docs/OV_Runtime_UG/supported_plugins/GPU.md @@ -56,13 +56,6 @@ Then, the device name can be passed to the ``ov::Core::compile_model()`` method, .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_default_gpu - .. tab-item:: Python :sync: py @@ -70,17 +63,17 @@ Then, the device name can be passed to the ``ov::Core::compile_model()`` method, :language: Python :fragment: compile_model_default_gpu + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_default_gpu + .. tab-item:: specific GPU .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_gpu_with_id - .. tab-item:: Python :sync: py @@ -88,17 +81,17 @@ Then, the device name can be passed to the ``ov::Core::compile_model()`` method, :language: Python :fragment: compile_model_gpu_with_id + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_gpu_with_id + .. tab-item:: specific tile .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_gpu_with_id_and_tile - .. tab-item:: Python :sync: py @@ -106,6 +99,13 @@ Then, the device name can be passed to the ``ov::Core::compile_model()`` method, :language: Python :fragment: compile_model_gpu_with_id_and_tile + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_gpu_with_id_and_tile + Supported Inference Data Types ####################################### @@ -149,13 +149,6 @@ It is done by specifying ``MULTI:GPU.1,GPU.0`` as a target device. .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_multi - .. tab-item:: Python :sync: py @@ -163,6 +156,13 @@ It is done by specifying ``MULTI:GPU.1,GPU.0`` as a target device. :language: Python :fragment: compile_model_multi + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_multi + For more details, see the :doc:`Multi-device execution `. @@ -180,13 +180,6 @@ Alternatively, it can be enabled explicitly via the device notion, for example ` .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_batch_plugin - .. tab-item:: Python :sync: py @@ -194,17 +187,17 @@ Alternatively, it can be enabled explicitly via the device notion, for example ` :language: Python :fragment: compile_model_batch_plugin + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_batch_plugin + .. tab-item:: Batching via throughput hint .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp - :language: cpp - :fragment: compile_model_auto_batch - .. tab-item:: Python :sync: py @@ -212,6 +205,13 @@ Alternatively, it can be enabled explicitly via the device notion, for example ` :language: Python :fragment: compile_model_auto_batch + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/compile_model.cpp + :language: cpp + :fragment: compile_model_auto_batch + For more details, see the :doc:`Automatic batching`. @@ -272,13 +272,6 @@ The code snippet below demonstrates examples of a bounded dynamic batch: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/dynamic_batch.cpp - :language: cpp - :fragment: dynamic_batch - .. tab-item:: Python :sync: py @@ -286,6 +279,13 @@ The code snippet below demonstrates examples of a bounded dynamic batch: :language: Python :fragment: dynamic_batch + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/dynamic_batch.cpp + :language: cpp + :fragment: dynamic_batch + Notes for performance and memory consumption in dynamic shapes -------------------------------------------------------------- @@ -348,13 +348,6 @@ The GPU plugin has the following additional preprocessing options: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes.cpp - :language: cpp - :fragment: init_preproc - .. tab-item:: Python :sync: py @@ -362,6 +355,13 @@ The GPU plugin has the following additional preprocessing options: :language: Python :fragment: init_preproc + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_two_planes.cpp + :language: cpp + :fragment: init_preproc + With such preprocessing, GPU plugin will expect ``ov::intel_gpu::ocl::ClImage2DTensor`` (or derived) to be passed for each NV12 plane via ``ov::InferRequest::set_tensor()`` or ``ov::InferRequest::set_tensors()`` methods. diff --git a/docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md b/docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md index cf675115abe..7847bcef2d4 100644 --- a/docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md +++ b/docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md @@ -51,32 +51,6 @@ of ``ov::RemoteContext`` derived classes. .. tab-set:: - .. tab-item:: Linux/C++ - :sync: linux-cpp - - .. tab-set:: - - .. tab-item:: Create from cl_context - :sync: create-from-cl-context - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp - :language: cpp - :fragment: [context_from_cl_context] - - .. tab-item:: Create from cl_queue - :sync: create-from-cl-queue - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp - :language: cpp - :fragment: [context_from_cl_queue] - - .. tab-item:: Create from VADisplay - :sync: create-from-vadisplay - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp - :language: cpp - :fragment: [context_from_va_display] - .. tab-item:: Windows/C++ :sync: windows-cpp @@ -102,6 +76,58 @@ of ``ov::RemoteContext`` derived classes. .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp :language: cpp :fragment: [context_from_d3d_device] + + .. tab-item:: Windows/C + :sync: windows-c + + .. tab-set:: + + .. tab-item:: Create from cl_context + :sync: create-from-cl-context + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp + :language: c + :fragment: [context_from_cl_context] + + .. tab-item:: Create from cl_queue + :sync: create-from-cl-queue + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp + :language: c + :fragment: [context_from_cl_queue] + + .. tab-item:: Create from ID3D11Device + :sync: create-from-id3d11device + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp + :language: c + :fragment: [context_from_d3d_device] + + .. tab-item:: Linux/C++ + :sync: linux-cpp + + .. tab-set:: + + .. tab-item:: Create from cl_context + :sync: create-from-cl-context + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp + :language: cpp + :fragment: [context_from_cl_context] + + .. tab-item:: Create from cl_queue + :sync: create-from-cl-queue + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp + :language: cpp + :fragment: [context_from_cl_queue] + + .. tab-item:: Create from VADisplay + :sync: create-from-vadisplay + + .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp + :language: cpp + :fragment: [context_from_va_display] .. tab-item:: Linux/C :sync: linux-c @@ -129,32 +155,6 @@ of ``ov::RemoteContext`` derived classes. :language: c :fragment: [context_from_va_display] - .. tab-item:: Windows/C - :sync: windows-c - - .. tab-set:: - - .. tab-item:: Create from cl_context - :sync: create-from-cl-context - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp - :language: c - :fragment: [context_from_cl_context] - - .. tab-item:: Create from cl_queue - :sync: create-from-cl-queue - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp - :language: c - :fragment: [context_from_cl_queue] - - .. tab-item:: Create from ID3D11Device - :sync: create-from-id3d11device - - .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp - :language: c - :fragment: [context_from_d3d_device] - Getting RemoteContext from the Plugin +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ @@ -173,14 +173,14 @@ To request the current default context of the plugin, use one of the following m .. tab-set:: .. tab-item:: Get context from Core - :sync: context-core + :sync: get-context-core .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp :language: cpp :fragment: [default_context_from_core] .. tab-item:: Get context from compiled model - :sync: context-compiled-model + :sync: get-context-compiled-model .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp :language: cpp @@ -192,14 +192,14 @@ To request the current default context of the plugin, use one of the following m .. tab-set:: .. tab-item:: Get context from Core - :sync: context-core + :sync: get-context-core .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp :language: c :fragment: [default_context_from_core] .. tab-item:: Get context from compiled model - :sync: context-compiled-model + :sync: get-context-compiled-model .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp :language: c @@ -255,7 +255,7 @@ For more details, see the code snippets below: :fragment: [wrap_cl_image] .. tab-item:: biplanar NV12 surface - :sync: biplanar + :sync: biplanar-nv12-surface .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation.cpp :language: cpp @@ -323,7 +323,7 @@ For more details, see the code snippets below: :fragment: [wrap_cl_image] .. tab-item:: biplanar NV12 surface - :sync: biplanar + :sync: biplanar-nv12-surface .. doxygensnippet:: docs/snippets/gpu/remote_objects_creation_c.cpp :language: c @@ -370,7 +370,7 @@ should be added before model compilation: .. tab-set:: .. tab-item:: two-plane - :sync: two_plane + :sync: two-plane .. tab-set:: @@ -389,7 +389,7 @@ should be added before model compilation: :fragment: [init_preproc] .. tab-item:: single-plane - :sync: single_plane + :sync: single-plane .. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_single_plane.cpp :language: cpp @@ -467,7 +467,7 @@ inputs need to be set via the ``ov::InferRequest::set_tensors`` method with vect :fragment: [batched_case] .. tab-item:: NV12 to Grey - :sync: single-plane + :sync: nv12-grey .. doxygensnippet:: docs/snippets/gpu/preprocessing_nv12_to_gray.cpp :language: cpp diff --git a/docs/OV_Runtime_UG/supported_plugins/config_properties.md b/docs/OV_Runtime_UG/supported_plugins/config_properties.md index f5671df8927..c6ca9c26817 100644 --- a/docs/OV_Runtime_UG/supported_plugins/config_properties.md +++ b/docs/OV_Runtime_UG/supported_plugins/config_properties.md @@ -43,13 +43,6 @@ devices enabled by OpenVINO plugins and returns information, using the ``ov::Cor .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [get_available_devices] - .. tab-item:: Python :sync: py @@ -57,6 +50,13 @@ devices enabled by OpenVINO plugins and returns information, using the ``ov::Cor :language: py :fragment: [get_available_devices] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [get_available_devices] + The function returns a list of available devices, for example: @@ -100,13 +100,6 @@ The code below demonstrates how to query ``HETERO`` device priority of devices w .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [hetero_priorities] - .. tab-item:: Python :sync: py @@ -114,6 +107,13 @@ The code below demonstrates how to query ``HETERO`` device priority of devices w :language: py :fragment: [hetero_priorities] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [hetero_priorities] + .. note:: @@ -125,13 +125,6 @@ supported properties (``ov::supported_properties``), and others, use the ``ov::C .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [cpu_device_name] - .. tab-item:: Python :sync: py @@ -139,6 +132,13 @@ supported properties (``ov::supported_properties``), and others, use the ``ov::C :language: py :fragment: [cpu_device_name] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [cpu_device_name] + A returned value appears as follows: ``Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz``. @@ -163,13 +163,6 @@ accept a selection of properties as last arguments. Each of the properties shoul .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [compile_model_with_property] - .. tab-item:: Python :sync: py @@ -177,6 +170,13 @@ accept a selection of properties as last arguments. Each of the properties shoul :language: py :fragment: [compile_model_with_property] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [compile_model_with_property] + The example below specifies hints that a model should be compiled to be inferred with multiple inference requests in parallel to achieve best throughput, while inference should be performed without accuracy loss with FP32 precision. @@ -191,13 +191,6 @@ However, setting properties on a specific ``ov::Core::compile_model`` call appli .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [core_set_property_then_compile] - .. tab-item:: Python :sync: py @@ -205,6 +198,13 @@ However, setting properties on a specific ``ov::Core::compile_model`` call appli :language: py :fragment: [core_set_property_then_compile] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [core_set_property_then_compile] + Properties on CompiledModel Level ########################################################### @@ -218,13 +218,6 @@ compiled model level property such as ``ov::optimal_number_of_infer_requests``: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [optimal_number_of_infer_requests] - .. tab-item:: Python :sync: py @@ -232,19 +225,19 @@ compiled model level property such as ``ov::optimal_number_of_infer_requests``: :language: py :fragment: [optimal_number_of_infer_requests] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [optimal_number_of_infer_requests] + Or the number of threads that would be used for inference on ``CPU`` device: .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [inference_num_threads] - .. tab-item:: Python :sync: py @@ -252,6 +245,13 @@ Or the number of threads that would be used for inference on ``CPU`` device: :language: py :fragment: [inference_num_threads] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [inference_num_threads] + Setting Properties for Compiled Model +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ @@ -261,13 +261,6 @@ The only mode that supports this method is [Multi-Device execution](../multi_dev .. tab-set:: - .. tab-item:: C++ - :sync: cpp - - .. doxygensnippet:: docs/snippets/ov_properties_api.cpp - :language: cpp - :fragment: [multi_device] - .. tab-item:: Python :sync: py @@ -275,6 +268,13 @@ The only mode that supports this method is [Multi-Device execution](../multi_dev :language: py :fragment: [multi_device] + .. tab-item:: C++ + :sync: cpp + + .. doxygensnippet:: docs/snippets/ov_properties_api.cpp + :language: cpp + :fragment: [multi_device] + @endsphinxdirective diff --git a/docs/get_started/get_started_demos.md b/docs/get_started/get_started_demos.md index cecb009be0f..077b4e9b8b1 100644 --- a/docs/get_started/get_started_demos.md +++ b/docs/get_started/get_started_demos.md @@ -105,6 +105,13 @@ This guide used the following model to run the Image Classification Sample: .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + omz_downloader --name googlenet-v1 --output_dir %USERPROFILE%\Documents\models + .. tab-item:: Linux :sync: linux @@ -112,15 +119,8 @@ This guide used the following model to run the Image Classification Sample: omz_downloader --name googlenet-v1 --output_dir ~/models - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - omz_downloader --name googlenet-v1 --output_dir %USERPROFILE%\Documents\models - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -131,6 +131,23 @@ This guide used the following model to run the Image Classification Sample: .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + ################|| Downloading models ||################ + + ========== Downloading C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.prototxt + ... 100%, 9 KB, ? KB/s, 0 seconds passed + + ========== Downloading C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel + ... 100%, 4834 KB, 571 KB/s, 8 seconds passed + + ################|| Post-processing ||################ + + ========== Replacing text in C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.prototxt + .. tab-item:: Linux :sync: linux @@ -147,25 +164,8 @@ This guide used the following model to run the Image Classification Sample: ========= Replacing text in /home/username/models/public/googlenet-v1/googlenet-v1.prototxt ========= - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - ################|| Downloading models ||################ - - ========== Downloading C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.prototxt - ... 100%, 9 KB, ? KB/s, 0 seconds passed - - ========== Downloading C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel - ... 100%, 4834 KB, 571 KB/s, 8 seconds passed - - ################|| Post-processing ||################ - - ========== Replacing text in C:\Users\username\Documents\models\public\googlenet-v1\googlenet-v1.prototxt - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -202,6 +202,13 @@ Create an ```` directory to contain the model's Intermediate Representat .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + mkdir %USERPROFILE%\Documents\ir + .. tab-item:: Linux :sync: linux @@ -209,15 +216,8 @@ Create an ```` directory to contain the model's Intermediate Representat mkdir ~/ir - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - mkdir %USERPROFILE%\Documents\ir - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -238,6 +238,13 @@ The command with most placeholders filled in and FP16 precision: .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + mo --input_model %USERPROFILE%\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel --compress_to_fp16 --output_dir %USERPROFILE%\Documents\ir + .. tab-item:: Linux :sync: linux @@ -245,15 +252,8 @@ The command with most placeholders filled in and FP16 precision: mo --input_model ~/models/public/googlenet-v1/googlenet-v1.caffemodel --compress_to_fp16 --output_dir ~/ir - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - mo --input_model %USERPROFILE%\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel --compress_to_fp16 --output_dir %USERPROFILE%\Documents\ir - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -282,6 +282,13 @@ To run the **Image Classification** code sample with an input image using the IR .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + \setupvars.bat + .. tab-item:: Linux :sync: linux @@ -289,15 +296,8 @@ To run the **Image Classification** code sample with an input image using the IR source /setupvars.sh - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - \setupvars.bat - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -307,6 +307,13 @@ To run the **Image Classification** code sample with an input image using the IR .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + cd %USERPROFILE%\Documents\Intel\OpenVINO\openvino_samples_build\intel64\Release + .. tab-item:: Linux :sync: linux @@ -314,15 +321,8 @@ To run the **Image Classification** code sample with an input image using the IR cd ~/openvino_cpp_samples_build/intel64/Release - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - cd %USERPROFILE%\Documents\Intel\OpenVINO\openvino_samples_build\intel64\Release - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -332,6 +332,13 @@ To run the **Image Classification** code sample with an input image using the IR .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + classification_sample_async.exe -i -m -d + .. tab-item:: Linux :sync: linux @@ -339,15 +346,8 @@ To run the **Image Classification** code sample with an input image using the IR classification_sample_async -i -m -d - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - classification_sample_async.exe -i -m -d - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -363,6 +363,13 @@ The following command shows how to run the Image Classification Code Sample usin .. tab-set:: + .. tab-item:: Windows + :sync: windows + + .. code-block:: bat + + .\classification_sample_async.exe -i %USERPROFILE%\Downloads\dog.bmp -m %USERPROFILE%\Documents\ir\googlenet-v1.xml -d CPU + .. tab-item:: Linux :sync: linux @@ -370,15 +377,8 @@ The following command shows how to run the Image Classification Code Sample usin ./classification_sample_async -i ~/Downloads/dog.bmp -m ~/ir/googlenet-v1.xml -d CPU - .. tab-item:: Windows - :sync: windows - - .. code-block:: bat - - .\classification_sample_async.exe -i %USERPROFILE%\Downloads\dog.bmp -m %USERPROFILE%\Documents\ir\googlenet-v1.xml -d CPU - .. tab-item:: macOS - :sync: mac + :sync: macos .. code-block:: sh @@ -418,13 +418,6 @@ Running Inference on GPU .. tab-set:: - .. tab-item:: Linux - :sync: linux - - .. code-block:: sh - - ./classification_sample_async -i ~/Downloads/dog.bmp -m ~/ir/googlenet-v1.xml -d GPU - .. tab-item:: Windows :sync: windows @@ -432,6 +425,13 @@ Running Inference on GPU .\classification_sample_async.exe -i %USERPROFILE%\Downloads\dog.bmp -m %USERPROFILE%\Documents\ir\googlenet-v1.xml -d GPU + .. tab-item:: Linux + :sync: linux + + .. code-block:: sh + + ./classification_sample_async -i ~/Downloads/dog.bmp -m ~/ir/googlenet-v1.xml -d GPU + Other Demos and Samples ####################### diff --git a/docs/install_guides/configurations-for-intel-gpu.md b/docs/install_guides/configurations-for-intel-gpu.md index dc30080a11f..c2ee2128b77 100644 --- a/docs/install_guides/configurations-for-intel-gpu.md +++ b/docs/install_guides/configurations-for-intel-gpu.md @@ -31,7 +31,7 @@ Depending on your operating system, there may be different methods to install th .. tab-set:: .. tab-item:: Ubuntu 22.04 LTS - :sync: ubuntu22 + :sync: ubuntu-22 Download and install the `deb` packages published `here `__ and install the apt package `ocl-icd-libopencl1` with the OpenCl ICD loader. @@ -42,7 +42,7 @@ Depending on your operating system, there may be different methods to install th apt-get install -y ocl-icd-libopencl1 intel-opencl-icd intel-level-zero-gpu level-zero .. tab-item:: Ubuntu 20.04 LTS - :sync: ubuntu20 + :sync: ubuntu-20 Ubuntu 20.04 LTS is not updated with the latest driver versions. You can install the updated versions up to the version 22.43 from apt: @@ -57,7 +57,7 @@ Depending on your operating system, there may be different methods to install th Alternatively, download older `deb` version from `here `__. Note that older driver version might not include some of the bug fixes and might be not supported on some latest platforms. Check the supported hardware for the versions you are installing. .. tab-item:: RedHat UBI 8 - :sync: redhat8 + :sync: redhat-8 Follow the `guide `__ to add Yum repository. diff --git a/docs/install_guides/installing-model-dev-tools.md b/docs/install_guides/installing-model-dev-tools.md index f9744368897..a0be0427286 100644 --- a/docs/install_guides/installing-model-dev-tools.md +++ b/docs/install_guides/installing-model-dev-tools.md @@ -70,19 +70,19 @@ Create a virtual Python environment to avoid dependency conflicts. To create a v .. tab-set:: - .. tab-item:: Linux and macOS - :sync: lin-mac - - .. code-block:: sh - - python3 -m venv openvino_env - .. tab-item:: Windows :sync: windows .. code-block:: sh python -m venv openvino_env + + .. tab-item:: Linux and macOS + :sync: linux-and-macos + + .. code-block:: sh + + python3 -m venv openvino_env @@ -93,13 +93,6 @@ Activate the newly created Python virtual environment by issuing this command: .. tab-set:: - .. tab-item:: Linux and macOS - :sync: lin-mac - - .. code-block:: sh - - source openvino_env/bin/activate - .. tab-item:: Windows :sync: windows @@ -107,6 +100,13 @@ Activate the newly created Python virtual environment by issuing this command: openvino_env\Scripts\activate + .. tab-item:: Linux and macOS + :sync: linux-and-macos + + .. code-block:: sh + + source openvino_env/bin/activate + .. important:: The above command must be re-run every time a new command terminal window is opened. diff --git a/docs/install_guides/installing-openvino-apt.md b/docs/install_guides/installing-openvino-apt.md index 9ed258ea5b4..5de39d868ee 100644 --- a/docs/install_guides/installing-openvino-apt.md +++ b/docs/install_guides/installing-openvino-apt.md @@ -35,7 +35,7 @@ Prerequisites .. tab-set:: .. tab-item:: System Requirements - :sync: sys-req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `__ @@ -47,7 +47,7 @@ Prerequisites See `Product Specifications `__ for information about your processor. .. tab-item:: Software Requirements - :sync: soft-req + :sync: software-requirements * `CMake 3.13 or higher, 64-bit `__ * GCC 7.5.0 (for Ubuntu 18.04), GCC 9.3.0 (for Ubuntu 20.04) or GCC 11.3.0 (for Ubuntu 22.04) @@ -91,21 +91,21 @@ Step 1: Set Up the OpenVINO Toolkit APT Repository .. tab-set:: .. tab-item:: Ubuntu 22 - :sync: ubuntu22 + :sync: ubuntu-22 .. code-block:: sh echo "deb https://apt.repos.intel.com/openvino/2023 ubuntu22 main" | sudo tee /etc/apt/sources.list.d/intel-openvino-2023.list .. tab-item:: Ubuntu 20 - :sync: ubuntu20 + :sync: ubuntu-20 .. code-block:: sh echo "deb https://apt.repos.intel.com/openvino/2023 ubuntu20 main" | sudo tee /etc/apt/sources.list.d/intel-openvino-2023.list .. tab-item:: Ubuntu 18 - :sync: ubuntu18 + :sync: ubuntu-18 .. code-block:: sh @@ -136,7 +136,7 @@ Install OpenVINO Runtime .. tab-set:: .. tab-item:: The Latest Version - :sync: latest + :sync: latest-version Run the following command: diff --git a/docs/install_guides/installing-openvino-brew.md b/docs/install_guides/installing-openvino-brew.md index 277012e02c2..a6d02d79e96 100644 --- a/docs/install_guides/installing-openvino-brew.md +++ b/docs/install_guides/installing-openvino-brew.md @@ -27,25 +27,16 @@ See `Installing Additional Components <#optional-installing-additional-component .. tab-set:: .. tab-item:: System Requirements - :sync: sys-req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `__ .. tab-item:: Software Requirements - :sync: soft-req + :sync: software-requirements .. tab-set:: - .. tab-item:: macOS - :sync: mac - - * `Homebrew `_ - * `CMake 3.13 or higher `__ (choose "macOS 10.13 or later"). Add ``/Applications/CMake.app/Contents/bin`` to path (for default installation). - * `Python 3.7 - 3.11 `__ . Install and add it to path. - * Apple Xcode Command Line Tools. In the terminal, run ``xcode-select --install`` from any directory to install it. - * (Optional) Apple Xcode IDE (not required for OpenVINO™, but useful for development) - .. tab-item:: Linux :sync: linux @@ -53,7 +44,16 @@ See `Installing Additional Components <#optional-installing-additional-component * `CMake 3.13 or higher, 64-bit `__ * GCC 7.5.0 (for Ubuntu 18.04), GCC 9.3.0 (for Ubuntu 20.04) or GCC 11.3.0 (for Ubuntu 22.04) * `Python 3.7 - 3.10, 64-bit `__ - + + .. tab-item:: macOS + :sync: macos + + * `Homebrew `_ + * `CMake 3.13 or higher `__ (choose "macOS 10.13 or later"). Add ``/Applications/CMake.app/Contents/bin`` to path (for default installation). + * `Python 3.7 - 3.11 `__ . Install and add it to path. + * Apple Xcode Command Line Tools. In the terminal, run ``xcode-select --install`` from any directory to install it. + * (Optional) Apple Xcode IDE (not required for OpenVINO™, but useful for development) + Installing OpenVINO Runtime diff --git a/docs/install_guides/installing-openvino-conda.md b/docs/install_guides/installing-openvino-conda.md index cf7f9bd04fd..571e88c2838 100644 --- a/docs/install_guides/installing-openvino-conda.md +++ b/docs/install_guides/installing-openvino-conda.md @@ -16,7 +16,7 @@ .. tab-set:: .. tab-item:: System Requirements - :sync: sys-req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `__ diff --git a/docs/install_guides/installing-openvino-from-archive-linux.md b/docs/install_guides/installing-openvino-from-archive-linux.md index b4b906226f7..b8d5a07e200 100644 --- a/docs/install_guides/installing-openvino-from-archive-linux.md +++ b/docs/install_guides/installing-openvino-from-archive-linux.md @@ -25,7 +25,7 @@ See the `Release Notes `__ @@ -35,7 +35,7 @@ See the `Release Notes `__ for information about your processor. .. tab-item:: Software - :sync: soft + :sync: software * `CMake 3.13 or higher, 64-bit `__ * `Python 3.7 - 3.11, 64-bit `__ @@ -43,23 +43,23 @@ See the `Release Notes `__ .. tab-item:: Software Requirements - :sync: soft-req + :sync: software-requirements * `CMake 3.13 or higher `__ (choose "macOS 10.13 or later"). Add ``/Applications/CMake.app/Contents/bin`` to path (for default install). * `Python 3.7 - 3.11 `__ (choose 3.7 - 3.11). Install and add to path. @@ -66,7 +66,7 @@ Step 1: Install OpenVINO Core Components .. tab-set:: .. tab-item:: x86, 64-bit - :sync: x86-64bit + :sync: x86-64 .. code-block:: sh @@ -75,7 +75,7 @@ Step 1: Install OpenVINO Core Components sudo mv m_openvino_toolkit_macos_10_15_2023.0.1.11005.fa1c41994f3_x86_64 /opt/intel/openvino_2023.0.1 .. tab-item:: ARM, 64-bit - :sync: arm-64bit + :sync: arm-64 .. code-block:: sh diff --git a/docs/install_guides/installing-openvino-from-archive-windows.md b/docs/install_guides/installing-openvino-from-archive-windows.md index 05ef83140f6..2f3bfc00ab3 100644 --- a/docs/install_guides/installing-openvino-from-archive-windows.md +++ b/docs/install_guides/installing-openvino-from-archive-windows.md @@ -25,7 +25,7 @@ System Requirements .. tab-set:: .. tab-item:: System Requirements - :sync: sys_req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `_ diff --git a/docs/install_guides/installing-openvino-pip.md b/docs/install_guides/installing-openvino-pip.md index 2932b764048..4ec3cbb2a2e 100644 --- a/docs/install_guides/installing-openvino-pip.md +++ b/docs/install_guides/installing-openvino-pip.md @@ -30,20 +30,20 @@ To create a virtual environment, use the following command: .. tab-set:: - .. tab-item:: Linux and macOS - :sync: lin-mac - - .. code-block:: sh - - python3 -m venv openvino_env - .. tab-item:: Windows - :sync: win + :sync: windows .. code-block:: sh python -m venv openvino_env + .. tab-item:: Linux and macOS + :sync: linux-and-macos + + .. code-block:: sh + + python3 -m venv openvino_env + Step 2. Activate Virtual Environment ++++++++++++++++++++++++++++++++++++ @@ -51,20 +51,20 @@ Step 2. Activate Virtual Environment .. tab-set:: - .. tab-item:: Linux and macOS - :sync: lin-mac - - .. code-block:: sh - - source openvino_env/bin/activate - .. tab-item:: Windows - :sync: win + :sync: windows .. code-block:: sh openvino_env\Scripts\activate + .. tab-item:: Linux and macOS + :sync: linux-and-macos + + .. code-block:: sh + + source openvino_env/bin/activate + .. important:: diff --git a/docs/install_guides/installing-openvino-raspbian-removedfromtoc.md b/docs/install_guides/installing-openvino-raspbian-removedfromtoc.md index 4274d96c7b7..9bdffe93189 100644 --- a/docs/install_guides/installing-openvino-raspbian-removedfromtoc.md +++ b/docs/install_guides/installing-openvino-raspbian-removedfromtoc.md @@ -14,13 +14,13 @@ Development and Target Systems .. tab-set:: .. tab-item:: System Requirements - :sync: sys-req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `_ .. tab-item:: Software Requirements - :sync: soft-req + :sync: software-requirements * CMake 3.10 or higher * Python 3.7 - 3.11 diff --git a/docs/install_guides/installing-openvino-yum.md b/docs/install_guides/installing-openvino-yum.md index b9d0c256d93..6dfa0dad731 100644 --- a/docs/install_guides/installing-openvino-yum.md +++ b/docs/install_guides/installing-openvino-yum.md @@ -34,7 +34,7 @@ Prerequisites .. tab-set:: .. tab-item:: System Requirements - :sync: sys-req + :sync: system-requirements | Full requirement listing is available in: | `System Requirements Page `__ diff --git a/docs/install_guides/troubleshooting-steps.md b/docs/install_guides/troubleshooting-steps.md index 7174a92de2d..9de51f0cb12 100644 --- a/docs/install_guides/troubleshooting-steps.md +++ b/docs/install_guides/troubleshooting-steps.md @@ -18,7 +18,7 @@ Check the versions of OpenVINO Runtime and Development Tools .. tab-set:: .. tab-item:: Python - :sync: mo-python-api + :sync: py .. code-block:: python @@ -26,7 +26,7 @@ Check the versions of OpenVINO Runtime and Development Tools ov_model = convert_model(version=True) .. tab-item:: CLI - :sync: cli-tool + :sync: cli .. code-block:: sh diff --git a/docs/install_guides/uninstalling-openvino.md b/docs/install_guides/uninstalling-openvino.md index bd9fdbf768d..f2560048ad1 100644 --- a/docs/install_guides/uninstalling-openvino.md +++ b/docs/install_guides/uninstalling-openvino.md @@ -32,10 +32,10 @@ If you have installed OpenVINO Runtime from archive files, you can uninstall it rmdir /s del - - + + .. tab-item:: Linux & macOS - :sync: lin-mac + :sync: linux-and-macos If you have created the symbolic link, remove the link first: @@ -49,6 +49,5 @@ If you have installed OpenVINO Runtime from archive files, you can uninstall it rm -r && rm - @endsphinxdirective diff --git a/docs/notebooks-installation.md b/docs/notebooks-installation.md index 836981f3bbe..d82347640fa 100644 --- a/docs/notebooks-installation.md +++ b/docs/notebooks-installation.md @@ -126,7 +126,7 @@ operating system or environment. sudo yum install python36-devel mesa-libGL .. tab-item:: macOS - :sync: mac + :sync: macos Alternatively, you may skip steps 1-3 if you prefer to manually install `Python 3 `__ and `Git `__. @@ -158,7 +158,7 @@ operating system or environment. If OpenVINO is installed globally, do not run any of these commands in a terminal where ``setupvars.sh`` is sourced. .. tab-item:: Azure ML - :sync: azure + :sync: azure-ml .. note:: @@ -302,7 +302,6 @@ operating system or environment. pip install -r requirements.txt - .. tab-item:: Linux :sync: linux @@ -341,10 +340,9 @@ operating system or environment. .. code-block:: sh pip install -r requirements.txt - .. tab-item:: macOS - :sync: mac + :sync: macos 1. **Create a Virtual Environment** @@ -383,7 +381,7 @@ operating system or environment. .. tab-item:: Azure ML - :sync: azure + :sync: azure-ml 1. Create a Conda environment @@ -564,7 +562,7 @@ To reactivate your environment, run: source openvino_env/bin/activate .. tab-item:: macOS - :sync: mac + :sync: macos .. code:: bash @@ -595,7 +593,7 @@ This operation is optional. However, if you want to remove your virtual environm rm -rf openvino_env .. tab-item:: macOS - :sync: mac + :sync: macos .. code:: bash diff --git a/docs/optimization_guide/dldt_deployment_optimization_tput.md b/docs/optimization_guide/dldt_deployment_optimization_tput.md index 2bc174148da..f61d3ed112a 100644 --- a/docs/optimization_guide/dldt_deployment_optimization_tput.md +++ b/docs/optimization_guide/dldt_deployment_optimization_tput.md @@ -37,6 +37,13 @@ In general, most throughput-oriented inference applications should: * Setup the configuration for the *device* (for example, as parameters of the ``ov::Core::compile_model``) via either previously introduced :doc:`low-level explicit options ` or :doc:`OpenVINO performance hints ` (**preferable**): .. tab-set:: + + .. tab-item:: Python + :sync: py + + .. doxygensnippet:: docs/snippets/ov_auto_batching.py + :language: python + :fragment: [compile_model] .. tab-item:: C++ :sync: cpp @@ -44,14 +51,6 @@ In general, most throughput-oriented inference applications should: .. doxygensnippet:: docs/snippets/ov_auto_batching.cpp :language: cpp :fragment: [compile_model] - - .. tab-item:: Python - :sync: py - - .. doxygensnippet:: docs/snippets/ov_auto_batching.py - :language: python - :fragment: [compile_model] - * Query the ``ov::optimal_number_of_infer_requests`` from the ``ov::CompiledModel`` (resulted from a compilation of the model for the device) to create the number of the requests required to saturate the device. diff --git a/docs/optimization_guide/nncf/filter_pruning.md b/docs/optimization_guide/nncf/filter_pruning.md index 4fd0d80de75..38a0f6bd527 100644 --- a/docs/optimization_guide/nncf/filter_pruning.md +++ b/docs/optimization_guide/nncf/filter_pruning.md @@ -47,7 +47,7 @@ In this step, NNCF-related imports are added in the beginning of the training sc :fragment: [imports] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -69,7 +69,7 @@ of optimization methods (`"compression"` section). :fragment: [nncf_congig] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -104,7 +104,7 @@ optimization. :fragment: [wrap_model] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -127,7 +127,7 @@ of the original model. :fragment: [tune_model] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -150,7 +150,7 @@ fine-tuning that will inform optimization methods to do some adjustments to func :fragment: [distributed] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -172,7 +172,7 @@ the case of PyTorch and frozen graph - for TensorFlow 2. :fragment: [export] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -198,7 +198,7 @@ To save model checkpoint use the following API: :fragment: [save_checkpoint] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -220,7 +220,7 @@ To restore the model from checkpoint you should use the following API: :fragment: [load_checkpoint] .. tab-item:: TensorFlow 2 - :sync: tf2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python diff --git a/docs/optimization_guide/nncf/ptq/basic_quantization_flow.md b/docs/optimization_guide/nncf/ptq/basic_quantization_flow.md index afb4f54e2fd..4399b6cdb88 100644 --- a/docs/optimization_guide/nncf/ptq/basic_quantization_flow.md +++ b/docs/optimization_guide/nncf/ptq/basic_quantization_flow.md @@ -36,7 +36,7 @@ The transformation function is a function that takes a sample from the dataset a .. tab-set:: .. tab-item:: OpenVINO - :sync: ov + :sync: openvino .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py :language: python @@ -57,7 +57,7 @@ The transformation function is a function that takes a sample from the dataset a :fragment: [dataset] .. tab-item:: TensorFlow - :sync: tf + :sync: tensorflow .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py :language: python @@ -74,7 +74,7 @@ Once the dataset is ready and the model object is instantiated, you can apply 8- .. tab-set:: .. tab-item:: OpenVINO - :sync: ov + :sync: openvino .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py :language: python @@ -95,7 +95,7 @@ Once the dataset is ready and the model object is instantiated, you can apply 8- :fragment: [quantization] .. tab-item:: TensorFlow - :sync: tf + :sync: tensorflow .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py :language: python @@ -107,7 +107,7 @@ After that the model can be converted into the OpenVINO Intermediate Representat .. tab-set:: .. tab-item:: OpenVINO - :sync: ov + :sync: openvino .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py :language: python @@ -128,7 +128,7 @@ After that the model can be converted into the OpenVINO Intermediate Representat :fragment: [inference] .. tab-item:: TensorFlow - :sync: tf + :sync: tensorflow .. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py :language: python diff --git a/docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md b/docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md index 466d0af431e..ba6e2c8fe88 100644 --- a/docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md +++ b/docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md @@ -24,7 +24,7 @@ This step is similar to the :doc:`Basic 8-bit quantization