From 5a50266dd758603f99b849d5e3f9043a1f5141a1 Mon Sep 17 00:00:00 2001 From: Sebastian Golebiewski Date: Wed, 9 Nov 2022 17:02:33 +0100 Subject: [PATCH] DOCS: Fixing formatting in Multi Device (#13292) --- docs/OV_Runtime_UG/multi_device.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/OV_Runtime_UG/multi_device.md b/docs/OV_Runtime_UG/multi_device.md index c239a85db15..eeb1d314393 100644 --- a/docs/OV_Runtime_UG/multi_device.md +++ b/docs/OV_Runtime_UG/multi_device.md @@ -4,14 +4,14 @@ To run inference on multiple devices, you can choose either of the following ways: -- Use the :ref:`CUMULATIVE_THROUGHPUT option ` of the Automatic Device Selection mode. This way, you can use all available devices in the system without the need to specify them, +- Use the :ref:`CUMULATIVE_THROUGHPUT option ` of the Automatic Device Selection mode. This way, you can use all available devices in the system without the need to specify them. - Use the Multi-Device execution mode. This page will explain how it works and how to use it. @endsphinxdirective ## How MULTI Works -The Multi-Device execution mode, or MULTI for short, acts as a "virtual" or a "proxy" device, which does not bind to a specific type of hardware. Instead, it assigns available computing devices to particular inference requests, which are then executed in parallel. +The Multi-Device execution mode, or MULTI for short, acts as a "virtual" or a "proxy" device, which does not bind to a specific type of hardware. Instead, it assigns available computing devices to particular inference requests, which are then executed in parallel. The potential gains from using Multi-Device execution are: * improved throughput from using multiple devices at once, @@ -22,7 +22,7 @@ Importantly, the Multi-Device mode does not change the application logic, so it Note that the performance increase in this mode comes from utilizing multiple devices at once. This means that you need to provide the devices with enough inference requests to keep them busy, otherwise you will not benefit much from using MULTI. -## Using the Multi-Device Mode +## Using the Multi-Device Mode Following the OpenVINO™ naming convention, the Multi-Device mode is assigned the label of “MULTI.” The only configuration option available for it is a prioritized list of devices to use: @@ -111,8 +111,8 @@ Alternatively, you can combine all the individual device settings into a single ### Querying the Optimal Number of Inference Requests -When using MULTI, you don't need to sum over included devices yourself, you can query the optimal number of requests directly, -using the [configure devices](supported_plugins/config_properties.md) property: +When using MULTI, you don't need to sum over included devices yourself, you can query the optimal number of requests directly, +using the [configure devices](supported_plugins/config_properties.md) property: @sphinxdirective @@ -128,14 +128,14 @@ using the [configure devices](supported_plugins/config_properties.md) property: ## Using the Multi-Device with OpenVINO Samples and Benchmarking Performance -To see how the Multi-Device execution is used in practice and test its performance, take a look at OpenVINO's Benchmark Application which presents the optimal performance of the plugin without the need for additional settings, like the number of requests or CPU threads. -Here is an example command to evaluate performance of HDDL+GPU: +To see how the Multi-Device execution is used in practice and test its performance, take a look at OpenVINO's Benchmark Application which presents the optimal performance of the plugin without the need for additional settings, like the number of requests or CPU threads. +Here is an example command to evaluate performance of HDDL+GPU: ```sh ./benchmark_app –d MULTI:HDDL,GPU –m -i -niter 1000 ``` -For more information, refer to the [C++](../../samples/cpp/benchmark_app/README.md) or [Python](../../tools/benchmark_tool/README.md) version instructions. +For more information, refer to the [C++](../../samples/cpp/benchmark_app/README.md) or [Python](../../tools/benchmark_tool/README.md) version instructions. @sphinxdirective .. note:: @@ -148,15 +148,15 @@ For more information, refer to the [C++](../../samples/cpp/benchmark_app/README. ## Performance Considerations for the Multi-Device Execution For best performance when using the MULTI execution mode you should consider a few recommendations: -- MULTI usually performs best when the fastest device is specified first in the device candidate list. -This is particularly important when the request-level parallelism is not sufficient +- MULTI usually performs best when the fastest device is specified first in the device candidate list. +This is particularly important when the request-level parallelism is not sufficient (e.g. the number of requests is not enough to saturate all devices). -- Just like with any throughput-oriented execution mode, it is highly recommended to query the optimal number of inference requests -directly from the instance of the `ov:compiled_model`. Refer to the code of the previously mentioned `benchmark_app` for more details. -- Execution on certain device combinations, for example CPU+GPU, performs better with certain knobs. Refer to the `benchmark_app` code for details. One specific example is disabling GPU driver polling, which in turn requires multiple GPU streams to balance out slower +- Just like with any throughput-oriented execution mode, it is highly recommended to query the optimal number of inference requests +directly from the instance of the `ov:compiled_model`. Refer to the code of the previously mentioned `benchmark_app` for more details. +- Execution on certain device combinations, for example CPU+GPU, performs better with certain knobs. Refer to the `benchmark_app` code for details. One specific example is disabling GPU driver polling, which in turn requires multiple GPU streams to balance out slower communication of inference completion from the device to the host. -- The MULTI logic always attempts to save on copying data between device-agnostic and user-facing inference requests, -and device-specific 'worker' requests that are being actually scheduled behind the scene. +- The MULTI logic always attempts to save on copying data between device-agnostic and user-facing inference requests, +and device-specific 'worker' requests that are being actually scheduled behind the scene. To facilitate the copy savings, it is recommended to run the requests in the order in which they were created. - While performance of accelerators combines well with MULTI, the CPU+GPU execution may introduce certain performance issues. It is due to the devices sharing some resources, like power or bandwidth. Enabling the GPU throttling hint, which saves a CPU thread for CPU inference, is an example of a recommended solution addressing this issue.