From 5393b8a77a671b82586f7122745da4299bf55ae2 Mon Sep 17 00:00:00 2001 From: Sebastian Golebiewski Date: Tue, 11 Jun 2024 12:08:07 +0200 Subject: [PATCH] [DOCS] Performance hint and thread scheduling update (#24946) Fixing reference and formatting. Proofreading. --- .../assets/snippets/multi_threading.py | 2 +- .../cpu-device.rst | 8 +- ...erformance-hint-and-thread-scheduling.rst} | 81 +++++++++++-------- 3 files changed, 53 insertions(+), 38 deletions(-) rename docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/{performance-hint-and-threads-scheduling.rst => performance-hint-and-thread-scheduling.rst} (74%) diff --git a/docs/articles_en/assets/snippets/multi_threading.py b/docs/articles_en/assets/snippets/multi_threading.py index 6994b26a0d6..edefa760318 100644 --- a/docs/articles_en/assets/snippets/multi_threading.py +++ b/docs/articles_en/assets/snippets/multi_threading.py @@ -37,7 +37,7 @@ compiled_model_3 = core.compile_model( # ! [ov:intel_cpu:multi_threading:part0] # ! [ov:intel_cpu:multi_threading:part1] -# Disable CPU threads pinning for inference when the system supports it +# Disable CPU thread pinning for inference when the system supports it compiled_model_4 = core.compile_model( model=model, device_name=device_name, diff --git a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst index d95f97959f5..e2f87fed9d3 100644 --- a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst +++ b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst @@ -7,7 +7,7 @@ CPU Device :maxdepth: 1 :hidden: - cpu-device/performance-hint-and-threads-scheduling + cpu-device/performance-hint-and-thread-scheduling .. meta:: :description: The CPU plugin in the Intel® Distribution of OpenVINO™ toolkit @@ -234,6 +234,8 @@ This can be achieved by specifying ``MULTI:CPU,GPU.0`` as a target device in cas For more details, see the :doc:`Multi-device execution ` article. +.. _multi_stream_execution: + Multi-stream Execution +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ @@ -242,7 +244,7 @@ property is set for CPU plugin, then multiple streams are created for the model. host thread, which means that incoming infer requests can be processed simultaneously. Each stream is pinned to its own group of physical cores with respect to NUMA nodes physical memory usage to minimize overhead on data transfer between NUMA nodes. -For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`threads scheduling introduction `. +For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`thread scheduling introduction `. .. note:: @@ -390,7 +392,7 @@ Multi-Threading Optimization CPU inference will infer an input or multiple inputs in parallel on multiple logical processors. -For more details, see the :doc:`threads scheduling introduction `. +For more details, see the :doc:`thread scheduling introduction `. Denormals Optimization diff --git a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-threads-scheduling.rst b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-thread-scheduling.rst similarity index 74% rename from docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-threads-scheduling.rst rename to docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-thread-scheduling.rst index 3087bcf2d95..5abd4dbcec6 100644 --- a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-threads-scheduling.rst +++ b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-thread-scheduling.rst @@ -1,48 +1,48 @@ - -Performance Hints and Threads Scheduling +Performance Hints and Thread Scheduling ======================================== .. meta:: - :description: The Threads Scheduling of CPU plugin in OpenVINO™ Runtime + :description: Thread Scheduling of the CPU plugin in OpenVINO™ Runtime detects CPU architecture and sets low-level properties based on performance hints automatically. -While all supported devices in OpenVINO offer low-level performance settings, it is advisable -not to use these settings widely unless targeting specific platforms and models. The recommended -approach is to configure performance in OpenVINO Runtime using the high-level performance hints -property ``ov::hint::performance_mode``. Performance hints ensure optimal portability and -scalability of applications across various platforms and models. - -To simplify the configuration of hardware devices, OpenVINO offers two performance hints: the -latency hint ``ov::hint::PerformanceMode::LATENCY`` and the throughput hint -``ov::hint::PerformanceMode::THROUGHPUT``. +To simplify the configuration of hardware devices, it is recommended to use the +:doc:` ov::hint::PerformanceMode::LATENCY and ov::hint::PerformanceMode::THROUGHPUT <../../../../optimize-inference/high-level-performance-hints>` +high-level performance hints. Both performance hints ensure optimal portability +and scalability of applications across various platforms and models. - ``ov::inference_num_threads`` limits the number of logical processors used for CPU inference. If the number set by the user is greater than the number of logical processors on the platform, the multi-threading scheduler only uses the platform number for CPU inference. - ``ov::num_streams`` limits the number of infer requests that can be run in parallel. If the number set by the user is greater than the number of inference threads, multi-threading - scheduler only uses the number of inference threads to ensure that there is at least one thread per stream. -- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when the user runs - inference on a hybird platform that includes both Performance-cores (P-cores) and Efficient-cores (E-cores). - If the user platform only has one type of CPU core, this property has no effect, and CPU inference always uses this unique core type. + scheduler only uses the number of inference threads to ensure that there is at least + one thread per stream. +- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when + the user runs inference on a hybird platform that includes both Performance-cores (P-cores) + and Efficient-cores (E-cores). If the user platform only has one type of CPU core, this + property has no effect, and CPU inference always uses this unique core type. - ``ov::hint::enable_hyper_threading`` limits the use of one or two logical processors per CPU core when the platform has CPU hyperthreading enabled. - If there is only one logical processor per CPU core, such as Efficient-cores, this property has no effect, and CPU inference uses all logical processors. + If there is only one logical processor per CPU core, such as Efficient-cores, this + property has no effect, and CPU inference uses all logical processors. - ``ov::hint::enable_cpu_pinning`` enables CPU pinning during CPU inference. - If the user enables this property but the inference scenario does not support it, this property will be disabled during model compilation. + If the user enables this property but the inference scenario does not support it, this + property will be disabled during model compilation. -For additional details on the above configurations, refer to `Multi-stream Execution `__. +For additional details on the above configurations, refer to +:ref:`Multi-stream Execution `. Latency Hint -################################### +##################### In this scenario, the default setting of ``ov::hint::scheduling_core_type`` is determined by the model precision and the ratio of P-cores and E-cores. .. note:: - P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These types of cores are available starting with the 12th Gen Intel® Core™ processors. + P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These + types of cores are available starting with the 12th Gen Intel® Core™ processors. .. _Core Type Table of Latency Hint: +----------------------------+---------------------+---------------------+ @@ -57,7 +57,8 @@ the model precision and the ratio of P-cores and E-cores. .. note:: - Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel® Core™ processors on Windows. + Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel® + Core™ processors on Windows. Then the default settings for low-level performance properties on Windows and Linux are as follows: @@ -77,13 +78,20 @@ Then the default settings for low-level performance properties on Windows and Li .. note:: - - ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a specific platform based on internal heuristics to guarantee optimal performance. - - Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors on Windows, except in the case of large language models. - - In case hyper-threading is enabled, two logical processors share the hardware resources of one CPU core. OpenVINO does not expect to use both logical processors in one stream for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to ``No`` in this scenario. - - ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and enabled on Linux. Such default settings are aligned with typical workloads running in the corresponding environments to guarantee better out-of-the-box (OOB) performance. + - ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a + specific platform based on internal heuristics to guarantee optimal performance. + - Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors + on Windows, except in the case of large language models. + - In case hyper-threading is enabled, two logical processors share the hardware resources + of one CPU core. OpenVINO does not expect to use both logical processors in one stream + for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to + ``No`` in this scenario. + - ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and + enabled on Linux. Such default settings are aligned with typical workloads running + in the corresponding environments to guarantee better out-of-the-box (OOB) performance. Throughput Hint -###################################### +##################### In this scenario, thread scheduling first evaluates the memory pressure of the model being inferred on the current platform, and determines the number of threads per stream, as shown below. @@ -99,7 +107,8 @@ inferred on the current platform, and determines the number of threads per strea +-----------------+-----------------------+ Then the value of ``ov::num_streams`` is calculated by dividing ``ov::inference_num_threads`` -by the number of threads per stream. The default settings for low-level performance properties on Windows and Linux are as follows: +by the number of threads per stream. The default settings for low-level performance +properties on Windows and Linux are as follows: +--------------------------------------+-------------------------------+-------------------------------+ | Property | Windows | Linux | @@ -117,13 +126,15 @@ by the number of threads per stream. The default settings for low-level performa .. note:: - - By default, different core types are not mixed within a single stream in this scenario. The cores from different NUMA nodes are not mixed within a single stream. + By default, different core types are not mixed within a single stream in this scenario. + The cores from different NUMA nodes are not mixed within a single stream. Multi-Threading Optimization -############################################## +############################ The following properties can be used to limit the available CPU resources for model inference. -If the platform or operating system supports this behavior, the OpenVINO Runtime will perform multi-threading scheduling based on the limited available CPU. +If the platform or operating system supports this behavior, the OpenVINO Runtime will +perform multi-threading scheduling based on the limited available CPU. - ``ov::inference_num_threads`` - ``ov::hint::scheduling_core_type`` @@ -148,11 +159,13 @@ If the platform or operating system supports this behavior, the OpenVINO Runtime .. note:: - ``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support Intel® x86-64 CPU on Linux and Windows in the current release. + ``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support + Intel® x86-64 CPU on Linux and Windows in the current release. In some use cases, OpenVINO Runtime will enable CPU thread pinning by default for better performance. Users can also turn this feature on or off using the property ``ov::hint::enable_cpu_pinning``. -Disabling thread pinning may be beneficial in complex applications where several workloads are executed in parallel. +Disabling thread pinning may be beneficial in complex applications where several workloads +are executed in parallel. .. tab-set:: @@ -172,4 +185,4 @@ Disabling thread pinning may be beneficial in complex applications where several For details on multi-stream execution check the -:doc:`optimization guide <../../optimize-inference/optimizing-throughput/advanced_throughput_options>`. \ No newline at end of file +:doc:`optimization guide <../../optimize-inference/optimizing-throughput/advanced_throughput_options>`.