natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] Performance hint and thread scheduling update (#24946) 

Fixing reference and formatting. Proofreading.
This commit is contained in:
Sebastian Golebiewski 2024-06-11 12:08:07 +02:00 committed by GitHub
parent deb5e63ff2
commit 5393b8a77a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 53 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/articles_en/assets/snippets/multi_threading.py
View File

@ -37,7 +37,7 @@ compiled_model_3 = core.compile_model(
# ! [ov:intel_cpu:multi_threading:part0]
# ! [ov:intel_cpu:multi_threading:part1]
# Disable CPU threads pinning for inference when the system supports it
# Disable CPU thread pinning for inference when the system supports it
compiled_model_4 = core.compile_model(
model=model,
device_name=device_name,

8
docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst
View File

@ -7,7 +7,7 @@ CPU Device
:maxdepth: 1
:hidden:
cpu-device/performance-hint-and-threads-scheduling
cpu-device/performance-hint-and-thread-scheduling
.. meta::
:description: The CPU plugin in the Intel® Distribution of OpenVINO™ toolkit
@ -234,6 +234,8 @@ This can be achieved by specifying ``MULTI:CPU,GPU.0`` as a target device in cas
For more details, see the :doc:`Multi-device execution <multi-device>` article.
.. _multi_stream_execution:
Multi-stream Execution
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@ -242,7 +244,7 @@ property is set for CPU plugin, then multiple streams are created for the model.
host thread, which means that incoming infer requests can be processed simultaneously. Each stream is pinned to its own group of
physical cores with respect to NUMA nodes physical memory usage to minimize overhead on data transfer between NUMA nodes.
For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`threads scheduling introduction <cpu-device/performance-hint-and-threads-scheduling>`.
For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`thread scheduling introduction <cpu-device/performance-hint-and-thread-scheduling>`.
.. note::
@ -390,7 +392,7 @@ Multi-Threading Optimization
CPU inference will infer an input or multiple inputs in parallel on multiple logical processors.
For more details, see the :doc:`threads scheduling introduction <cpu-device/performance-hint-and-threads-scheduling>`.
For more details, see the :doc:`thread scheduling introduction <cpu-device/performance-hint-and-thread-scheduling>`.
Denormals Optimization

81
docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-threads-scheduling.rst → docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/performance-hint-and-thread-scheduling.rst
View File

@ -1,48 +1,48 @@
Performance Hints and Threads Scheduling
Performance Hints and Thread Scheduling
========================================
.. meta::
:description: The Threads Scheduling of CPU plugin in OpenVINO™ Runtime
:description: Thread Scheduling of the CPU plugin in OpenVINO™ Runtime
detects CPU architecture and sets low-level properties based
on performance hints automatically.
While all supported devices in OpenVINO offer low-level performance settings, it is advisable
not to use these settings widely unless targeting specific platforms and models. The recommended
approach is to configure performance in OpenVINO Runtime using the high-level performance hints
property ``ov::hint::performance_mode``. Performance hints ensure optimal portability and
scalability of applications across various platforms and models.
To simplify the configuration of hardware devices, OpenVINO offers two performance hints: the
latency hint ``ov::hint::PerformanceMode::LATENCY`` and the throughput hint
``ov::hint::PerformanceMode::THROUGHPUT``.
To simplify the configuration of hardware devices, it is recommended to use the
:doc:` ov::hint::PerformanceMode::LATENCY and ov::hint::PerformanceMode::THROUGHPUT <../../../../optimize-inference/high-level-performance-hints>`
high-level performance hints. Both performance hints ensure optimal portability
and scalability of applications across various platforms and models.
- ``ov::inference_num_threads`` limits the number of logical processors used for CPU inference.
If the number set by the user is greater than the number of logical processors on the platform,
the multi-threading scheduler only uses the platform number for CPU inference.
- ``ov::num_streams`` limits the number of infer requests that can be run in parallel.
If the number set by the user is greater than the number of inference threads, multi-threading
scheduler only uses the number of inference threads to ensure that there is at least one thread per stream.
- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when the user runs
inference on a hybird platform that includes both Performance-cores (P-cores) and Efficient-cores (E-cores).
If the user platform only has one type of CPU core, this property has no effect, and CPU inference always uses this unique core type.
scheduler only uses the number of inference threads to ensure that there is at least
one thread per stream.
- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when
the user runs inference on a hybird platform that includes both Performance-cores (P-cores)
and Efficient-cores (E-cores). If the user platform only has one type of CPU core, this
property has no effect, and CPU inference always uses this unique core type.
- ``ov::hint::enable_hyper_threading`` limits the use of one or two logical processors per CPU
core when the platform has CPU hyperthreading enabled.
If there is only one logical processor per CPU core, such as Efficient-cores, this property has no effect, and CPU inference uses all logical processors.
If there is only one logical processor per CPU core, such as Efficient-cores, this
property has no effect, and CPU inference uses all logical processors.
- ``ov::hint::enable_cpu_pinning`` enables CPU pinning during CPU inference.
If the user enables this property but the inference scenario does not support it, this property will be disabled during model compilation.
If the user enables this property but the inference scenario does not support it, this
property will be disabled during model compilation.
For additional details on the above configurations, refer to `Multi-stream Execution <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.html#multi-stream-execution>`__.
For additional details on the above configurations, refer to
:ref:`Multi-stream Execution <multi_stream_execution>`.
Latency Hint
###################################
#####################
In this scenario, the default setting of ``ov::hint::scheduling_core_type`` is determined by
the model precision and the ratio of P-cores and E-cores.
.. note::
P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These types of cores are available starting with the 12th Gen Intel® Core™ processors.
P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These
types of cores are available starting with the 12th Gen Intel® Core™ processors.
.. _Core Type Table of Latency Hint:
+----------------------------+---------------------+---------------------+
@ -57,7 +57,8 @@ the model precision and the ratio of P-cores and E-cores.
.. note::
Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel® Core™ processors on Windows.
Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel®
Core™ processors on Windows.
Then the default settings for low-level performance properties on Windows and Linux are as follows:
@ -77,13 +78,20 @@ Then the default settings for low-level performance properties on Windows and Li
.. note::
- ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a specific platform based on internal heuristics to guarantee optimal performance.
- Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors on Windows, except in the case of large language models.
- In case hyper-threading is enabled, two logical processors share the hardware resources of one CPU core. OpenVINO does not expect to use both logical processors in one stream for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to ``No`` in this scenario.
- ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and enabled on Linux. Such default settings are aligned with typical workloads running in the corresponding environments to guarantee better out-of-the-box (OOB) performance.
- ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a
specific platform based on internal heuristics to guarantee optimal performance.
- Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors
on Windows, except in the case of large language models.
- In case hyper-threading is enabled, two logical processors share the hardware resources
of one CPU core. OpenVINO does not expect to use both logical processors in one stream
for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to
``No`` in this scenario.
- ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and
enabled on Linux. Such default settings are aligned with typical workloads running
in the corresponding environments to guarantee better out-of-the-box (OOB) performance.
Throughput Hint
######################################
#####################
In this scenario, thread scheduling first evaluates the memory pressure of the model being
inferred on the current platform, and determines the number of threads per stream, as shown below.
@ -99,7 +107,8 @@ inferred on the current platform, and determines the number of threads per strea
+-----------------+-----------------------+
Then the value of ``ov::num_streams`` is calculated by dividing ``ov::inference_num_threads``
by the number of threads per stream. The default settings for low-level performance properties on Windows and Linux are as follows:
by the number of threads per stream. The default settings for low-level performance
properties on Windows and Linux are as follows:
+--------------------------------------+-------------------------------+-------------------------------+
| Property | Windows | Linux |
@ -117,13 +126,15 @@ by the number of threads per stream. The default settings for low-level performa
.. note::
- By default, different core types are not mixed within a single stream in this scenario. The cores from different NUMA nodes are not mixed within a single stream.
By default, different core types are not mixed within a single stream in this scenario.
The cores from different NUMA nodes are not mixed within a single stream.
Multi-Threading Optimization
##############################################
############################
The following properties can be used to limit the available CPU resources for model inference.
If the platform or operating system supports this behavior, the OpenVINO Runtime will perform multi-threading scheduling based on the limited available CPU.
If the platform or operating system supports this behavior, the OpenVINO Runtime will
perform multi-threading scheduling based on the limited available CPU.
- ``ov::inference_num_threads``
- ``ov::hint::scheduling_core_type``
@ -148,11 +159,13 @@ If the platform or operating system supports this behavior, the OpenVINO Runtime
.. note::
``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support Intel® x86-64 CPU on Linux and Windows in the current release.
``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support
Intel® x86-64 CPU on Linux and Windows in the current release.
In some use cases, OpenVINO Runtime will enable CPU thread pinning by default for better performance.
Users can also turn this feature on or off using the property ``ov::hint::enable_cpu_pinning``.
Disabling thread pinning may be beneficial in complex applications where several workloads are executed in parallel.
Disabling thread pinning may be beneficial in complex applications where several workloads
are executed in parallel.
.. tab-set::
@ -172,4 +185,4 @@ Disabling thread pinning may be beneficial in complex applications where several
For details on multi-stream execution check the
:doc:`optimization guide <../../optimize-inference/optimizing-throughput/advanced_throughput_options>`.
:doc:`optimization guide <../../optimize-inference/optimizing-throughput/advanced_throughput_options>`.