[DOCS] postrelease adjustments (#23364)
This commit is contained in:
Karol Blaszczak
GitHub
parent
3084d7331b
commit
d64b4b4ade
|
|
@ -8,7 +8,6 @@ Compatibility and Support
|
|||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
compatibility-and-support/supported-models
|
||||
compatibility-and-support/supported-devices
|
||||
compatibility-and-support/supported-operations-inference-devices
|
||||
compatibility-and-support/supported-operations-framework-frontend
|
||||
|
|
@ -16,8 +15,6 @@ Compatibility and Support
|
|||
|
||||
:doc:`Supported Devices <compatibility-and-support/supported-devices>` - compatibility information for supported hardware accelerators.
|
||||
|
||||
:doc:`Supported Models <compatibility-and-support/supported-models>` - a table of models officially supported by OpenVINO.
|
||||
|
||||
:doc:`Supported Operations <compatibility-and-support/supported-operations-inference-devices>` - a listing of framework layers supported by OpenVINO.
|
||||
|
||||
:doc:`Supported Operations <compatibility-and-support/supported-operations-framework-frontend>` - a listing of layers supported by OpenVINO inference devices.
|
||||
|
|
|
|||
|
|
@ -1,36 +0,0 @@
|
|||
.. {#openvino_supported_models}
|
||||
|
||||
Supported Models
|
||||
================
|
||||
|
||||
|
||||
|
||||
.. meta::
|
||||
:description: Check the list of officially supported models in Intel®
|
||||
Distribution of OpenVINO™ toolkit.
|
||||
|
||||
|
||||
The OpenVINO team continues the effort to support as many models out-of-the-box as possible.
|
||||
Based on our research and user feedback, we prioritize the most common models and test them
|
||||
before every release. These models are considered officially supported.
|
||||
|
||||
|
||||
.. button-link:: ../../_static/download/OV_2023_models_supported.pdf
|
||||
:color: primary
|
||||
:outline:
|
||||
|
||||
:material-regular:`download;1.5em` Click for supported models [PDF]
|
||||
|
||||
The list is based on release 2023.0, as of June 01, 2023
|
||||
|
||||
| Note that the list provided here does not include all models supported by OpenVINO.
|
||||
| If your model is not included but is similar to those that are, it is still very likely to work.
|
||||
If your model fails to execute properly there are a few options available:
|
||||
|
||||
|
||||
* You can create a GitHub request for the operation(s) that are missing. These requests are reviewed regularly. You will be informed if and how the request will be accommodated. Additionally, your request may trigger a reply from someone in the community who can help.
|
||||
* As OpenVINO™ is open source you can enhance it with your own contribution to the GitHub repository. To learn more, see the articles on :doc:`OpenVINO Extensibility <../../documentation/openvino-extensibility>`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -3,7 +3,6 @@
|
|||
Performance Benchmarks
|
||||
======================
|
||||
|
||||
|
||||
.. meta::
|
||||
:description: Use the benchmark results for Intel® Distribution of OpenVINO™
|
||||
toolkit, that may help you decide what hardware to use or how
|
||||
|
|
@ -109,8 +108,9 @@ Please visit the tabs below for more information on key performance indicators a
|
|||
* input token length: 1024 (the tokens for GenAI models are in English).
|
||||
|
||||
|
||||
Platforms, Configurations, Methodology
|
||||
###########################################################
|
||||
.. raw:: html
|
||||
|
||||
<h2>Platforms, Configurations, Methodology</h2>
|
||||
|
||||
For a listing of all platforms and configurations used for testing, refer to the following:
|
||||
|
||||
|
|
@ -177,8 +177,9 @@ only to measuring performance.
|
|||
|
||||
|
||||
|
||||
Test performance yourself
|
||||
####################################
|
||||
.. raw:: html
|
||||
|
||||
<h2>Test performance yourself</h2>
|
||||
|
||||
You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers <performance-benchmarks/getting-performance-numbers>`.
|
||||
|
||||
|
|
@ -188,15 +189,16 @@ To learn more about it, visit `the website <https://www.intel.com/content/www/us
|
|||
or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h2>Disclaimers</h2>
|
||||
|
||||
Disclaimers
|
||||
####################################
|
||||
|
||||
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2024.0, as of March 06, 2024.
|
||||
|
||||
* OpenVINO Model Server performance results are based on release 2023.3, as of February 13, 2024.
|
||||
|
||||
The results may not reflect all publicly available updates. Intel technologies’ features and benefits depend on system configuration
|
||||
The results may not reflect all publicly available updates. Intel technologies' features and benefits depend on system configuration
|
||||
and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
|
||||
|
||||
See configuration disclosure for details. No product can be absolutely secure.
|
||||
|
|
@ -205,3 +207,18 @@ Your costs and results may vary.
|
|||
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<link rel="stylesheet" type="text/css" href="../_static/css/benchmark-banner.css">
|
||||
|
||||
.. container:: benchmark-banner
|
||||
|
||||
Results may vary. For more information, see
|
||||
:doc:`F.A.Q. <./performance-benchmarks/performance-benchmarks-faq>`
|
||||
See :doc:`Legal Information <./additional-resources/legal-information>`.
|
||||
|
|
@ -9,8 +9,10 @@ This guide explains how to use the benchmark_app to get performance numbers. It
|
|||
numbers are reflected through internal inference performance counters and execution graphs. It also includes
|
||||
information on using ITT and Intel® VTune™ Profiler to get performance insights.
|
||||
|
||||
Test performance with the benchmark_app
|
||||
###########################################################
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h2>Test performance with the benchmark_app</h2>
|
||||
|
||||
|
||||
|
||||
|
|
@ -22,8 +24,11 @@ Make sure to install the latest release package with support for frameworks of t
|
|||
For the most reliable performance benchmarks, :doc:`prepare the model for use with OpenVINO <../../openvino-workflow/model-preparation>`.
|
||||
|
||||
|
||||
Running the benchmark application
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>Running the benchmark application</h3>
|
||||
|
||||
|
||||
The benchmark_app includes a lot of device-specific options, but the primary usage is as simple as:
|
||||
|
||||
|
|
@ -46,11 +51,17 @@ it is recommended to always start performance evaluation with the :doc:`OpenVINO
|
|||
benchmark_app -hint latency -m <model> -d <device>
|
||||
|
||||
|
||||
Additional benchmarking considerations
|
||||
###########################################################
|
||||
|
||||
1 - Select a Proper Set of Operations to Measure
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
.. raw:: html
|
||||
|
||||
<h2>Additional benchmarking considerations</h2>
|
||||
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>1 - Select a Proper Set of Operations to Measure</h3>
|
||||
|
||||
|
||||
When evaluating performance of a model with OpenVINO Runtime, it is required to measure a proper set of operations.
|
||||
|
||||
|
|
@ -65,8 +76,10 @@ When evaluating performance of a model with OpenVINO Runtime, it is required to
|
|||
:doc:`General Runtime Optimizations <../../openvino-workflow/running-inference/optimize-inference/general-optimizations>`.
|
||||
|
||||
|
||||
2 - Try to Get Credible Data
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>2 - Try to Get Credible Data</h3>
|
||||
|
||||
Performance conclusions should be build upon reproducible data. As for the performance measurements, they should
|
||||
be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly
|
||||
|
|
@ -80,8 +93,10 @@ slower than the subsequent ones, an aggregated value can be used for the executi
|
|||
However, the end-to-end (application) benchmarking should also be performed under real operational conditions.
|
||||
|
||||
|
||||
3 - Compare Performance with Native/Framework Code
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>3 - Compare Performance with Native/Framework Code</h3>
|
||||
|
||||
When comparing the OpenVINO Runtime performance with the framework or another reference code, make sure that both versions are as similar as possible:
|
||||
|
||||
|
|
@ -92,8 +107,10 @@ When comparing the OpenVINO Runtime performance with the framework or another re
|
|||
- When applicable, leverage the :doc:`Dynamic Shapes support <../../openvino-workflow/running-inference/dynamic-shapes>`.
|
||||
- If possible, demand the same accuracy. For example, TensorFlow allows ``FP16`` execution, so when comparing to that, make sure to test the OpenVINO Runtime with the ``FP16`` as well.
|
||||
|
||||
Internal Inference Performance Counters and Execution Graphs
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>Internal Inference Performance Counters and Execution Graphs</h3>
|
||||
|
||||
More detailed insights into inference performance breakdown can be achieved with device-specific performance counters and/or execution graphs.
|
||||
Both :doc:`C++ and Python <../../learn-openvino/openvino-samples/benchmark-tool>`
|
||||
|
|
@ -145,8 +162,10 @@ Lastly, the performance statistics with both performance counters and execution
|
|||
so such data for the :doc:`inputs of dynamic shapes <../../openvino-workflow/running-inference/dynamic-shapes>` should be measured carefully,
|
||||
preferably by isolating the specific shape and executing multiple times in a loop, to gather reliable data.
|
||||
|
||||
Use ITT to Get Performance Insights
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<h3>Use ITT to Get Performance Insights</h3>
|
||||
|
||||
In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT).
|
||||
Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like
|
||||
|
|
@ -156,3 +175,13 @@ insights in the application-level performance on the timeline view.
|
|||
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<link rel="stylesheet" type="text/css" href="../_static/css/benchmark-banner.css">
|
||||
|
||||
.. container:: benchmark-banner
|
||||
|
||||
Results may vary. For more information, see
|
||||
:doc:`F.A.Q. <./performance-benchmarks-faq>` and
|
||||
:doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`.
|
||||
See :doc:`Legal Information <../additional-resources/legal-information>`.
|
||||
|
|
@ -5,10 +5,10 @@ Model Accuracy
|
|||
|
||||
|
||||
|
||||
The following two tables present the absolute accuracy drop calculated as the accuracy difference
|
||||
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
|
||||
FP16 representations of a model on three platform architectures. Please also refer to notes below
|
||||
the table for more information.
|
||||
The following two tables present the absolute accuracy drop calculated as the accuracy difference
|
||||
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
|
||||
FP16 representations of a model on three platform architectures. Please also refer to notes below
|
||||
the table for more information.
|
||||
|
||||
* A - Intel® Core™ i9-9000K (AVX2), INT8 and FP32
|
||||
* B - Intel® Xeon® 6338, (VNNI), INT8 and FP32
|
||||
|
|
@ -106,31 +106,31 @@ the table for more information.
|
|||
* - chatGLM2-6b
|
||||
- lambada openai
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 0.75
|
||||
- 0.75
|
||||
-
|
||||
-
|
||||
* - Llama-2-7b-chat
|
||||
- Wiki, StackExch, Crawl
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 3.38
|
||||
- 3.27
|
||||
-
|
||||
-
|
||||
* - Stable-Diffusion-V2-1
|
||||
- LIAON-5B
|
||||
- CLIP
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
* - Mistral-7b
|
||||
- proprietary Mistral.ai
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 3.49
|
||||
- 3.19
|
||||
-
|
||||
-
|
||||
|
||||
.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
|
||||
:header-rows: 1
|
||||
|
|
@ -234,37 +234,49 @@ the table for more information.
|
|||
* - chatGLM2-6b
|
||||
- lambada openai
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 0.75
|
||||
- 0.8
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
* - Llama-2-7b-chat
|
||||
- Wiki, StackExch, Crawl
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 3.26
|
||||
- 3.26
|
||||
-
|
||||
-
|
||||
-
|
||||
* - Stable-Diffusion-V2-1
|
||||
- LIAON-5B
|
||||
- CLIP
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
* - Mistral-7b
|
||||
- proprietary Mistral.ai
|
||||
- ppl
|
||||
-
|
||||
-
|
||||
- 3.18
|
||||
- 3.19
|
||||
-
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
|
||||
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
|
||||
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
|
||||
accuracy for the model.
|
||||
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
|
||||
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
|
||||
accuracy for the model.
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<link rel="stylesheet" type="text/css" href="../_static/css/benchmark-banner.css">
|
||||
|
||||
.. container:: benchmark-banner
|
||||
|
||||
Results may vary. For more information, see
|
||||
:doc:`F.A.Q. <./performance-benchmarks-faq>` and
|
||||
:doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`.
|
||||
See :doc:`Legal Information <../additional-resources/legal-information>`.
|
||||
|
|
@ -159,3 +159,14 @@ Performance Information F.A.Q.
|
|||
for autonomous vehicles, where a quick response to the result of the
|
||||
inference is required.
|
||||
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<link rel="stylesheet" type="text/css" href="../_static/css/benchmark-banner.css">
|
||||
|
||||
.. container:: benchmark-banner
|
||||
|
||||
Results may vary. For more information, see
|
||||
:doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`.
|
||||
See :doc:`Legal Information <../additional-resources/legal-information>`.
|
||||
|
|
@ -228,7 +228,7 @@ The following notebooks have been updated or newly added:
|
|||
* `Depth estimation with DepthAnything <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/280-depth-anything>`__
|
||||
* `Kosmos-2 <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/281-kosmos2-multimodal-large-language-model>`__
|
||||
* `Zero-shot Image Classification with SigLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/282-siglip-zero-shot-image-classification>`__
|
||||
* `Personalized image generation with PhotMaker <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/283-photo-maker>`__
|
||||
* `Personalized image generation with PhotoMaker <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/283-photo-maker>`__
|
||||
* `Voice tone cloning with OpenVoice <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/284-openvoice>`__
|
||||
* `Line-level text detection with Surya <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/285-surya-line-level-text-detection>`__
|
||||
* `InstantID: Zero-shot Identity-Preserving Generation using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/286-instant-id>`__
|
||||
|
|
@ -252,7 +252,7 @@ Known issues
|
|||
| *ID* - 132376
|
||||
| *Description:*
|
||||
| First-inference latency slow down for LLMs on Intel® Core™ Ultra processors. Up to 10-20%
|
||||
drop may occur due to radical memory optimization for processing ling sequences
|
||||
drop may occur due to radical memory optimization for processing long sequences
|
||||
(about 1.5-2 GB reduced memory usage).
|
||||
|
||||
| **Component - CPU runtime**
|
||||
|
|
|
|||
|
|
@ -130,6 +130,5 @@ Additional Resources
|
|||
====================
|
||||
|
||||
* `OpenVINO Success Stories <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/success-stories.html>`__ - See how Intel partners have successfully used OpenVINO in production applications to solve real-world problems.
|
||||
* :doc:`OpenVINO Supported Models <about-openvino/compatibility-and-support/supported-models>` - Check which models OpenVINO supports on your hardware.
|
||||
* :doc:`Performance Benchmarks <about-openvino/performance-benchmarks>` - View results from benchmarking models with OpenVINO on Intel hardware.
|
||||
|
||||
|
|
|
|||
|
|
@ -56,7 +56,8 @@ Install OpenVINO™ 2024.0
|
|||
NPU V\* V\* V\* n/a n/a n/a n/a V\*
|
||||
=============== ========== ====== ========= ======== ============ ========== ========== ==========
|
||||
|
||||
\* **Of the Linux systems, only Ubuntu 22.04 includes drivers for NPU device.**
|
||||
| \* **Of the Linux systems, only Ubuntu 22.04 includes drivers for NPU device.**
|
||||
| **For Windows, CPU inference on ARM64 is not supported.**
|
||||
|
||||
| **Build OpenVINO from source**
|
||||
| OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
|
||||
|
|
|
|||
|
|
@ -18,15 +18,20 @@ For an in-depth description of CPU plugin, see:
|
|||
- `OpenVINO Runtime CPU plugin source files <https://github.com/openvinotoolkit/openvino/tree/master/src/plugins/intel_cpu/>`__.
|
||||
|
||||
.. note::
|
||||
The scope of the CPU plugin features and optimizations on Arm® may differ from Intel® x86-64. If the limitation is not mentioned explicitly, the feature is supported for all CPU architectures.
|
||||
|
||||
The scope of the CPU plugin features and optimizations on Arm® may differ from
|
||||
Intel® x86-64. If the limitation is not mentioned explicitly, the feature is supported for
|
||||
all CPU architectures. **CPU inference on ARM64 is not supported for Windows.**
|
||||
|
||||
|
||||
Device Name
|
||||
###########################################################
|
||||
|
||||
The ``CPU`` device name is used for the CPU plugin. Even though there can be more than one physical socket on a platform, only one device of this kind is listed by OpenVINO.
|
||||
On multi-socket platforms, load balancing and memory usage distribution between NUMA nodes are handled automatically.
|
||||
In order to use CPU for inference, the device name should be passed to the ``ov::Core::compile_model()`` method:
|
||||
The ``CPU`` device name is used for the CPU plugin. Even though there can be more than one
|
||||
physical socket on a platform, only one device of this kind is listed by OpenVINO.
|
||||
On multi-socket platforms, load balancing and memory usage distribution between NUMA nodes are
|
||||
handled automatically. In order to use CPU for inference, the device name should be passed to
|
||||
the ``ov::Core::compile_model()`` method:
|
||||
|
||||
|
||||
.. tab-set::
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ OpenVINO 2024
|
|||
<ul class="splide__list">
|
||||
<li class="splide__slide">An open-source toolkit for optimizing and deploying deep learning models.<br>Boost your AI deep-learning inference performance!</li>
|
||||
|
||||
<li class="splide__slide"Better OpenVINO integration with PyTorch!<br>Use PyTorch models directly, without converting them first.<br>
|
||||
<li class="splide__slide">Better OpenVINO integration with PyTorch!<br>Use PyTorch models directly, without converting them first.<br>
|
||||
<a href="https://docs.openvino.ai/2024/openvino-workflow/model-preparation/convert-model-pytorch.html">Learn more...</a>
|
||||
</li>
|
||||
<li class="splide__slide">OpenVINO via PyTorch 2.0 torch.compile()<br>Use OpenVINO directly in PyTorch-native applications!<br>
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
/* the bottom banner only for benchmark pages */
|
||||
/* ========================================== */
|
||||
|
||||
.benchmark-banner {
|
||||
position: sticky;
|
||||
bottom: 0;
|
||||
width: 100%;
|
||||
z-index: 1001;
|
||||
text-align: center;
|
||||
padding: 7px;
|
||||
border-width: 0px;
|
||||
background: #525252;
|
||||
}
|
||||
.benchmark-banner p {
|
||||
margin: 0;
|
||||
color: #E9E9E9;
|
||||
}
|
||||
.benchmark-banner p a {
|
||||
color: #B4F0FF;
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Loading…
Reference in New Issue