[DOCS] Performance hint and thread scheduling update (#24946)
Fixing reference and formatting. Proofreading.
This commit is contained in:
Sebastian Golebiewski
GitHub
parent
deb5e63ff2
commit
5393b8a77a
|
|
@ -37,7 +37,7 @@ compiled_model_3 = core.compile_model(
|
|||
# ! [ov:intel_cpu:multi_threading:part0]
|
||||
|
||||
# ! [ov:intel_cpu:multi_threading:part1]
|
||||
# Disable CPU threads pinning for inference when the system supports it
|
||||
# Disable CPU thread pinning for inference when the system supports it
|
||||
compiled_model_4 = core.compile_model(
|
||||
model=model,
|
||||
device_name=device_name,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ CPU Device
|
|||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
cpu-device/performance-hint-and-threads-scheduling
|
||||
cpu-device/performance-hint-and-thread-scheduling
|
||||
|
||||
.. meta::
|
||||
:description: The CPU plugin in the Intel® Distribution of OpenVINO™ toolkit
|
||||
|
|
@ -234,6 +234,8 @@ This can be achieved by specifying ``MULTI:CPU,GPU.0`` as a target device in cas
|
|||
|
||||
For more details, see the :doc:`Multi-device execution <multi-device>` article.
|
||||
|
||||
.. _multi_stream_execution:
|
||||
|
||||
Multi-stream Execution
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
|
|
@ -242,7 +244,7 @@ property is set for CPU plugin, then multiple streams are created for the model.
|
|||
host thread, which means that incoming infer requests can be processed simultaneously. Each stream is pinned to its own group of
|
||||
physical cores with respect to NUMA nodes physical memory usage to minimize overhead on data transfer between NUMA nodes.
|
||||
|
||||
For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`threads scheduling introduction <cpu-device/performance-hint-and-threads-scheduling>`.
|
||||
For more details, see the :doc:`optimization guide <../optimize-inference>` and :doc:`thread scheduling introduction <cpu-device/performance-hint-and-thread-scheduling>`.
|
||||
|
||||
.. note::
|
||||
|
||||
|
|
@ -390,7 +392,7 @@ Multi-Threading Optimization
|
|||
|
||||
CPU inference will infer an input or multiple inputs in parallel on multiple logical processors.
|
||||
|
||||
For more details, see the :doc:`threads scheduling introduction <cpu-device/performance-hint-and-threads-scheduling>`.
|
||||
For more details, see the :doc:`thread scheduling introduction <cpu-device/performance-hint-and-thread-scheduling>`.
|
||||
|
||||
|
||||
Denormals Optimization
|
||||
|
|
|
|||
|
|
@ -1,48 +1,48 @@
|
|||
|
||||
Performance Hints and Threads Scheduling
|
||||
Performance Hints and Thread Scheduling
|
||||
========================================
|
||||
|
||||
.. meta::
|
||||
:description: The Threads Scheduling of CPU plugin in OpenVINO™ Runtime
|
||||
:description: Thread Scheduling of the CPU plugin in OpenVINO™ Runtime
|
||||
detects CPU architecture and sets low-level properties based
|
||||
on performance hints automatically.
|
||||
|
||||
While all supported devices in OpenVINO offer low-level performance settings, it is advisable
|
||||
not to use these settings widely unless targeting specific platforms and models. The recommended
|
||||
approach is to configure performance in OpenVINO Runtime using the high-level performance hints
|
||||
property ``ov::hint::performance_mode``. Performance hints ensure optimal portability and
|
||||
scalability of applications across various platforms and models.
|
||||
|
||||
To simplify the configuration of hardware devices, OpenVINO offers two performance hints: the
|
||||
latency hint ``ov::hint::PerformanceMode::LATENCY`` and the throughput hint
|
||||
``ov::hint::PerformanceMode::THROUGHPUT``.
|
||||
To simplify the configuration of hardware devices, it is recommended to use the
|
||||
:doc:` ov::hint::PerformanceMode::LATENCY and ov::hint::PerformanceMode::THROUGHPUT <../../../../optimize-inference/high-level-performance-hints>`
|
||||
high-level performance hints. Both performance hints ensure optimal portability
|
||||
and scalability of applications across various platforms and models.
|
||||
|
||||
- ``ov::inference_num_threads`` limits the number of logical processors used for CPU inference.
|
||||
If the number set by the user is greater than the number of logical processors on the platform,
|
||||
the multi-threading scheduler only uses the platform number for CPU inference.
|
||||
- ``ov::num_streams`` limits the number of infer requests that can be run in parallel.
|
||||
If the number set by the user is greater than the number of inference threads, multi-threading
|
||||
scheduler only uses the number of inference threads to ensure that there is at least one thread per stream.
|
||||
- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when the user runs
|
||||
inference on a hybird platform that includes both Performance-cores (P-cores) and Efficient-cores (E-cores).
|
||||
If the user platform only has one type of CPU core, this property has no effect, and CPU inference always uses this unique core type.
|
||||
scheduler only uses the number of inference threads to ensure that there is at least
|
||||
one thread per stream.
|
||||
- ``ov::hint::scheduling_core_type`` specifies the type of CPU cores for CPU inference when
|
||||
the user runs inference on a hybird platform that includes both Performance-cores (P-cores)
|
||||
and Efficient-cores (E-cores). If the user platform only has one type of CPU core, this
|
||||
property has no effect, and CPU inference always uses this unique core type.
|
||||
- ``ov::hint::enable_hyper_threading`` limits the use of one or two logical processors per CPU
|
||||
core when the platform has CPU hyperthreading enabled.
|
||||
If there is only one logical processor per CPU core, such as Efficient-cores, this property has no effect, and CPU inference uses all logical processors.
|
||||
If there is only one logical processor per CPU core, such as Efficient-cores, this
|
||||
property has no effect, and CPU inference uses all logical processors.
|
||||
- ``ov::hint::enable_cpu_pinning`` enables CPU pinning during CPU inference.
|
||||
If the user enables this property but the inference scenario does not support it, this property will be disabled during model compilation.
|
||||
If the user enables this property but the inference scenario does not support it, this
|
||||
property will be disabled during model compilation.
|
||||
|
||||
For additional details on the above configurations, refer to `Multi-stream Execution <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.html#multi-stream-execution>`__.
|
||||
For additional details on the above configurations, refer to
|
||||
:ref:`Multi-stream Execution <multi_stream_execution>`.
|
||||
|
||||
Latency Hint
|
||||
###################################
|
||||
#####################
|
||||
|
||||
In this scenario, the default setting of ``ov::hint::scheduling_core_type`` is determined by
|
||||
the model precision and the ratio of P-cores and E-cores.
|
||||
|
||||
.. note::
|
||||
|
||||
P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These types of cores are available starting with the 12th Gen Intel® Core™ processors.
|
||||
P-cores is short for Performance-cores and E-cores stands for Efficient-cores. These
|
||||
types of cores are available starting with the 12th Gen Intel® Core™ processors.
|
||||
|
||||
.. _Core Type Table of Latency Hint:
|
||||
+----------------------------+---------------------+---------------------+
|
||||
|
|
@ -57,7 +57,8 @@ the model precision and the ratio of P-cores and E-cores.
|
|||
|
||||
.. note::
|
||||
|
||||
Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel® Core™ processors on Windows.
|
||||
Both P-cores and E-cores may be used for any configuration starting with 14th Gen Intel®
|
||||
Core™ processors on Windows.
|
||||
|
||||
Then the default settings for low-level performance properties on Windows and Linux are as follows:
|
||||
|
||||
|
|
@ -77,13 +78,20 @@ Then the default settings for low-level performance properties on Windows and Li
|
|||
|
||||
.. note::
|
||||
|
||||
- ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a specific platform based on internal heuristics to guarantee optimal performance.
|
||||
- Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors on Windows, except in the case of large language models.
|
||||
- In case hyper-threading is enabled, two logical processors share the hardware resources of one CPU core. OpenVINO does not expect to use both logical processors in one stream for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to ``No`` in this scenario.
|
||||
- ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and enabled on Linux. Such default settings are aligned with typical workloads running in the corresponding environments to guarantee better out-of-the-box (OOB) performance.
|
||||
- ``ov::hint::scheduling_core_type`` may be adjusted for a particular inferred model on a
|
||||
specific platform based on internal heuristics to guarantee optimal performance.
|
||||
- Both P-cores and E-cores are used for the Latency Hint on Intel® Core™ Ultra Processors
|
||||
on Windows, except in the case of large language models.
|
||||
- In case hyper-threading is enabled, two logical processors share the hardware resources
|
||||
of one CPU core. OpenVINO does not expect to use both logical processors in one stream
|
||||
for a single infer request. So ``ov::hint::enable_hyper_threading`` is set to
|
||||
``No`` in this scenario.
|
||||
- ``ov::hint::enable_cpu_pinning`` is disabled by default on Windows and macOS, and
|
||||
enabled on Linux. Such default settings are aligned with typical workloads running
|
||||
in the corresponding environments to guarantee better out-of-the-box (OOB) performance.
|
||||
|
||||
Throughput Hint
|
||||
######################################
|
||||
#####################
|
||||
|
||||
In this scenario, thread scheduling first evaluates the memory pressure of the model being
|
||||
inferred on the current platform, and determines the number of threads per stream, as shown below.
|
||||
|
|
@ -99,7 +107,8 @@ inferred on the current platform, and determines the number of threads per strea
|
|||
+-----------------+-----------------------+
|
||||
|
||||
Then the value of ``ov::num_streams`` is calculated by dividing ``ov::inference_num_threads``
|
||||
by the number of threads per stream. The default settings for low-level performance properties on Windows and Linux are as follows:
|
||||
by the number of threads per stream. The default settings for low-level performance
|
||||
properties on Windows and Linux are as follows:
|
||||
|
||||
+--------------------------------------+-------------------------------+-------------------------------+
|
||||
| Property | Windows | Linux |
|
||||
|
|
@ -117,13 +126,15 @@ by the number of threads per stream. The default settings for low-level performa
|
|||
|
||||
.. note::
|
||||
|
||||
- By default, different core types are not mixed within a single stream in this scenario. The cores from different NUMA nodes are not mixed within a single stream.
|
||||
By default, different core types are not mixed within a single stream in this scenario.
|
||||
The cores from different NUMA nodes are not mixed within a single stream.
|
||||
|
||||
Multi-Threading Optimization
|
||||
##############################################
|
||||
############################
|
||||
|
||||
The following properties can be used to limit the available CPU resources for model inference.
|
||||
If the platform or operating system supports this behavior, the OpenVINO Runtime will perform multi-threading scheduling based on the limited available CPU.
|
||||
If the platform or operating system supports this behavior, the OpenVINO Runtime will
|
||||
perform multi-threading scheduling based on the limited available CPU.
|
||||
|
||||
- ``ov::inference_num_threads``
|
||||
- ``ov::hint::scheduling_core_type``
|
||||
|
|
@ -148,11 +159,13 @@ If the platform or operating system supports this behavior, the OpenVINO Runtime
|
|||
|
||||
.. note::
|
||||
|
||||
``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support Intel® x86-64 CPU on Linux and Windows in the current release.
|
||||
``ov::hint::scheduling_core_type`` and ``ov::hint::enable_hyper_threading`` only support
|
||||
Intel® x86-64 CPU on Linux and Windows in the current release.
|
||||
|
||||
In some use cases, OpenVINO Runtime will enable CPU thread pinning by default for better performance.
|
||||
Users can also turn this feature on or off using the property ``ov::hint::enable_cpu_pinning``.
|
||||
Disabling thread pinning may be beneficial in complex applications where several workloads are executed in parallel.
|
||||
Disabling thread pinning may be beneficial in complex applications where several workloads
|
||||
are executed in parallel.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
Loading…
Reference in New Issue