diff --git a/docs/articles_en/about-openvino/compatibility-and-support.rst b/docs/articles_en/about-openvino/compatibility-and-support.rst index 480a53f5639..f2c467be346 100644 --- a/docs/articles_en/about-openvino/compatibility-and-support.rst +++ b/docs/articles_en/about-openvino/compatibility-and-support.rst @@ -8,7 +8,6 @@ Compatibility and Support :maxdepth: 1 :hidden: - compatibility-and-support/supported-models compatibility-and-support/supported-devices compatibility-and-support/supported-operations-inference-devices compatibility-and-support/supported-operations-framework-frontend @@ -16,8 +15,6 @@ Compatibility and Support :doc:`Supported Devices ` - compatibility information for supported hardware accelerators. -:doc:`Supported Models ` - a table of models officially supported by OpenVINO. - :doc:`Supported Operations ` - a listing of framework layers supported by OpenVINO. :doc:`Supported Operations ` - a listing of layers supported by OpenVINO inference devices. diff --git a/docs/articles_en/about-openvino/compatibility-and-support/supported-models.rst b/docs/articles_en/about-openvino/compatibility-and-support/supported-models.rst deleted file mode 100644 index e25c74869e2..00000000000 --- a/docs/articles_en/about-openvino/compatibility-and-support/supported-models.rst +++ /dev/null @@ -1,36 +0,0 @@ -.. {#openvino_supported_models} - -Supported Models -================ - - - -.. meta:: - :description: Check the list of officially supported models in Intel® - Distribution of OpenVINO™ toolkit. - - -The OpenVINO team continues the effort to support as many models out-of-the-box as possible. -Based on our research and user feedback, we prioritize the most common models and test them -before every release. These models are considered officially supported. - - -.. button-link:: ../../_static/download/OV_2023_models_supported.pdf - :color: primary - :outline: - - :material-regular:`download;1.5em` Click for supported models [PDF] - -The list is based on release 2023.0, as of June 01, 2023 - -| Note that the list provided here does not include all models supported by OpenVINO. -| If your model is not included but is similar to those that are, it is still very likely to work. - If your model fails to execute properly there are a few options available: - - -* You can create a GitHub request for the operation(s) that are missing. These requests are reviewed regularly. You will be informed if and how the request will be accommodated. Additionally, your request may trigger a reply from someone in the community who can help. -* As OpenVINO™ is open source you can enhance it with your own contribution to the GitHub repository. To learn more, see the articles on :doc:`OpenVINO Extensibility <../../documentation/openvino-extensibility>`. - - - - diff --git a/docs/articles_en/about-openvino/performance-benchmarks.rst b/docs/articles_en/about-openvino/performance-benchmarks.rst index fa9ef966ad1..db8968f289b 100644 --- a/docs/articles_en/about-openvino/performance-benchmarks.rst +++ b/docs/articles_en/about-openvino/performance-benchmarks.rst @@ -3,7 +3,6 @@ Performance Benchmarks ====================== - .. meta:: :description: Use the benchmark results for Intel® Distribution of OpenVINO™ toolkit, that may help you decide what hardware to use or how @@ -109,8 +108,9 @@ Please visit the tabs below for more information on key performance indicators a * input token length: 1024 (the tokens for GenAI models are in English). -Platforms, Configurations, Methodology -########################################################### +.. raw:: html + +

Platforms, Configurations, Methodology

For a listing of all platforms and configurations used for testing, refer to the following: @@ -177,8 +177,9 @@ only to measuring performance. -Test performance yourself -#################################### +.. raw:: html + +

Test performance yourself

You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers `. @@ -188,15 +189,16 @@ To learn more about it, visit `the website `__. +.. raw:: html + +

Disclaimers

-Disclaimers -#################################### * Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2024.0, as of March 06, 2024. * OpenVINO Model Server performance results are based on release 2023.3, as of February 13, 2024. -The results may not reflect all publicly available updates. Intel technologies’ features and benefits depend on system configuration +The results may not reflect all publicly available updates. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer. See configuration disclosure for details. No product can be absolutely secure. @@ -205,3 +207,18 @@ Your costs and results may vary. Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products. + + + + + + +.. raw:: html + + + +.. container:: benchmark-banner + + Results may vary. For more information, see + :doc:`F.A.Q. <./performance-benchmarks/performance-benchmarks-faq>` + See :doc:`Legal Information <./additional-resources/legal-information>`. \ No newline at end of file diff --git a/docs/articles_en/about-openvino/performance-benchmarks/getting-performance-numbers.rst b/docs/articles_en/about-openvino/performance-benchmarks/getting-performance-numbers.rst index 5da9873da8b..cb7a204ebca 100644 --- a/docs/articles_en/about-openvino/performance-benchmarks/getting-performance-numbers.rst +++ b/docs/articles_en/about-openvino/performance-benchmarks/getting-performance-numbers.rst @@ -9,8 +9,10 @@ This guide explains how to use the benchmark_app to get performance numbers. It numbers are reflected through internal inference performance counters and execution graphs. It also includes information on using ITT and Intel® VTune™ Profiler to get performance insights. -Test performance with the benchmark_app -########################################################### + +.. raw:: html + +

Test performance with the benchmark_app

@@ -22,8 +24,11 @@ Make sure to install the latest release package with support for frameworks of t For the most reliable performance benchmarks, :doc:`prepare the model for use with OpenVINO <../../openvino-workflow/model-preparation>`. -Running the benchmark application -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +.. raw:: html + +

Running the benchmark application

+ The benchmark_app includes a lot of device-specific options, but the primary usage is as simple as: @@ -46,11 +51,17 @@ it is recommended to always start performance evaluation with the :doc:`OpenVINO benchmark_app -hint latency -m -d -Additional benchmarking considerations -########################################################### -1 - Select a Proper Set of Operations to Measure -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +.. raw:: html + +

Additional benchmarking considerations

+ + + +.. raw:: html + +

1 - Select a Proper Set of Operations to Measure

+ When evaluating performance of a model with OpenVINO Runtime, it is required to measure a proper set of operations. @@ -65,8 +76,10 @@ When evaluating performance of a model with OpenVINO Runtime, it is required to :doc:`General Runtime Optimizations <../../openvino-workflow/running-inference/optimize-inference/general-optimizations>`. -2 - Try to Get Credible Data -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +.. raw:: html + +

2 - Try to Get Credible Data

Performance conclusions should be build upon reproducible data. As for the performance measurements, they should be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly @@ -80,8 +93,10 @@ slower than the subsequent ones, an aggregated value can be used for the executi However, the end-to-end (application) benchmarking should also be performed under real operational conditions. -3 - Compare Performance with Native/Framework Code -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +.. raw:: html + +

3 - Compare Performance with Native/Framework Code

When comparing the OpenVINO Runtime performance with the framework or another reference code, make sure that both versions are as similar as possible: @@ -92,8 +107,10 @@ When comparing the OpenVINO Runtime performance with the framework or another re - When applicable, leverage the :doc:`Dynamic Shapes support <../../openvino-workflow/running-inference/dynamic-shapes>`. - If possible, demand the same accuracy. For example, TensorFlow allows ``FP16`` execution, so when comparing to that, make sure to test the OpenVINO Runtime with the ``FP16`` as well. -Internal Inference Performance Counters and Execution Graphs -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +.. raw:: html + +

Internal Inference Performance Counters and Execution Graphs

More detailed insights into inference performance breakdown can be achieved with device-specific performance counters and/or execution graphs. Both :doc:`C++ and Python <../../learn-openvino/openvino-samples/benchmark-tool>` @@ -145,8 +162,10 @@ Lastly, the performance statistics with both performance counters and execution so such data for the :doc:`inputs of dynamic shapes <../../openvino-workflow/running-inference/dynamic-shapes>` should be measured carefully, preferably by isolating the specific shape and executing multiple times in a loop, to gather reliable data. -Use ITT to Get Performance Insights -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +.. raw:: html + +

Use ITT to Get Performance Insights

In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT). Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like @@ -156,3 +175,13 @@ insights in the application-level performance on the timeline view. +.. raw:: html + + + +.. container:: benchmark-banner + + Results may vary. For more information, see + :doc:`F.A.Q. <./performance-benchmarks-faq>` and + :doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`. + See :doc:`Legal Information <../additional-resources/legal-information>`. \ No newline at end of file diff --git a/docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst b/docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst index 13d055f4cf5..de6ffe4fd60 100644 --- a/docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst +++ b/docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst @@ -5,10 +5,10 @@ Model Accuracy -The following two tables present the absolute accuracy drop calculated as the accuracy difference -between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and -FP16 representations of a model on three platform architectures. Please also refer to notes below -the table for more information. +The following two tables present the absolute accuracy drop calculated as the accuracy difference +between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and +FP16 representations of a model on three platform architectures. Please also refer to notes below +the table for more information. * A - Intel® Core™ i9-9000K (AVX2), INT8 and FP32 * B - Intel® Xeon® 6338, (VNNI), INT8 and FP32 @@ -106,31 +106,31 @@ the table for more information. * - chatGLM2-6b - lambada openai - ppl - - + - - 0.75 - 0.75 - - + - * - Llama-2-7b-chat - Wiki, StackExch, Crawl - ppl - - + - - 3.38 - 3.27 - - + - * - Stable-Diffusion-V2-1 - LIAON-5B - CLIP - - - - - - - + - + - + - * - Mistral-7b - proprietary Mistral.ai - ppl - - + - - 3.49 - 3.19 - - + - .. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only) :header-rows: 1 @@ -234,37 +234,49 @@ the table for more information. * - chatGLM2-6b - lambada openai - ppl - - + - - 0.75 - 0.8 - - - - + - + - * - Llama-2-7b-chat - Wiki, StackExch, Crawl - ppl - - + - - 3.26 - 3.26 - - + - - * - Stable-Diffusion-V2-1 - LIAON-5B - CLIP - - - - - - - - - - + - + - + - + - + - * - Mistral-7b - proprietary Mistral.ai - ppl - - + - - 3.18 - 3.19 - - - - + - + - -Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop. -For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured -accuracy for the model. +Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop. +For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured +accuracy for the model. + + +.. raw:: html + + + +.. container:: benchmark-banner + + Results may vary. For more information, see + :doc:`F.A.Q. <./performance-benchmarks-faq>` and + :doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`. + See :doc:`Legal Information <../additional-resources/legal-information>`. \ No newline at end of file diff --git a/docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst b/docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst index 8d20b839047..aed03386e69 100644 --- a/docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst +++ b/docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst @@ -159,3 +159,14 @@ Performance Information F.A.Q. for autonomous vehicles, where a quick response to the result of the inference is required. + + +.. raw:: html + + + +.. container:: benchmark-banner + + Results may vary. For more information, see + :doc:`Platforms, Configurations, Methodology <../performance-benchmarks>`. + See :doc:`Legal Information <../additional-resources/legal-information>`. \ No newline at end of file diff --git a/docs/articles_en/about-openvino/release-notes-openvino.rst b/docs/articles_en/about-openvino/release-notes-openvino.rst index 0c2280cbd02..c8b6b2dc342 100644 --- a/docs/articles_en/about-openvino/release-notes-openvino.rst +++ b/docs/articles_en/about-openvino/release-notes-openvino.rst @@ -228,7 +228,7 @@ The following notebooks have been updated or newly added: * `Depth estimation with DepthAnything `__ * `Kosmos-2 `__ * `Zero-shot Image Classification with SigLIP `__ -* `Personalized image generation with PhotMaker `__ +* `Personalized image generation with PhotoMaker `__ * `Voice tone cloning with OpenVoice `__ * `Line-level text detection with Surya `__ * `InstantID: Zero-shot Identity-Preserving Generation using OpenVINO `__ @@ -252,7 +252,7 @@ Known issues | *ID* - 132376 | *Description:* | First-inference latency slow down for LLMs on Intel® Core™ Ultra processors. Up to 10-20% - drop may occur due to radical memory optimization for processing ling sequences + drop may occur due to radical memory optimization for processing long sequences (about 1.5-2 GB reduced memory usage). | **Component - CPU runtime** diff --git a/docs/articles_en/get-started.rst b/docs/articles_en/get-started.rst index 05c316d7c4e..9f5261a7b12 100644 --- a/docs/articles_en/get-started.rst +++ b/docs/articles_en/get-started.rst @@ -130,6 +130,5 @@ Additional Resources ==================== * `OpenVINO Success Stories `__ - See how Intel partners have successfully used OpenVINO in production applications to solve real-world problems. -* :doc:`OpenVINO Supported Models ` - Check which models OpenVINO supports on your hardware. * :doc:`Performance Benchmarks ` - View results from benchmarking models with OpenVINO on Intel hardware. diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index c7c652a19f4..dbc9a844fa4 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -56,7 +56,8 @@ Install OpenVINO™ 2024.0 NPU V\* V\* V\* n/a n/a n/a n/a V\* =============== ========== ====== ========= ======== ============ ========== ========== ========== - \* **Of the Linux systems, only Ubuntu 22.04 includes drivers for NPU device.** + | \* **Of the Linux systems, only Ubuntu 22.04 includes drivers for NPU device.** + | **For Windows, CPU inference on ARM64 is not supported.** | **Build OpenVINO from source** | OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform, diff --git a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst index d933446c3a6..a85da611901 100644 --- a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst +++ b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst @@ -18,15 +18,20 @@ For an in-depth description of CPU plugin, see: - `OpenVINO Runtime CPU plugin source files `__. .. note:: - The scope of the CPU plugin features and optimizations on Arm® may differ from Intel® x86-64. If the limitation is not mentioned explicitly, the feature is supported for all CPU architectures. + + The scope of the CPU plugin features and optimizations on Arm® may differ from + Intel® x86-64. If the limitation is not mentioned explicitly, the feature is supported for + all CPU architectures. **CPU inference on ARM64 is not supported for Windows.** Device Name ########################################################### -The ``CPU`` device name is used for the CPU plugin. Even though there can be more than one physical socket on a platform, only one device of this kind is listed by OpenVINO. -On multi-socket platforms, load balancing and memory usage distribution between NUMA nodes are handled automatically. -In order to use CPU for inference, the device name should be passed to the ``ov::Core::compile_model()`` method: +The ``CPU`` device name is used for the CPU plugin. Even though there can be more than one +physical socket on a platform, only one device of this kind is listed by OpenVINO. +On multi-socket platforms, load balancing and memory usage distribution between NUMA nodes are +handled automatically. In order to use CPU for inference, the device name should be passed to +the ``ov::Core::compile_model()`` method: .. tab-set:: diff --git a/docs/home.rst b/docs/home.rst index 1b820446b1b..6812ee27f3f 100644 --- a/docs/home.rst +++ b/docs/home.rst @@ -24,7 +24,7 @@ OpenVINO 2024
  • An open-source toolkit for optimizing and deploying deep learning models.
    Boost your AI deep-learning inference performance!
    • -
  • Use PyTorch models directly, without converting them first.
    +
  • Better OpenVINO integration with PyTorch!
    Use PyTorch models directly, without converting them first.
    Learn more...
  • OpenVINO via PyTorch 2.0 torch.compile()
    Use OpenVINO directly in PyTorch-native applications!
    diff --git a/docs/sphinx_setup/_static/css/benchmark-banner.css b/docs/sphinx_setup/_static/css/benchmark-banner.css new file mode 100644 index 00000000000..5e96ced9f54 --- /dev/null +++ b/docs/sphinx_setup/_static/css/benchmark-banner.css @@ -0,0 +1,20 @@ +/* the bottom banner only for benchmark pages */ +/* ========================================== */ + +.benchmark-banner { + position: sticky; + bottom: 0; + width: 100%; + z-index: 1001; + text-align: center; + padding: 7px; + border-width: 0px; + background: #525252; +} +.benchmark-banner p { + margin: 0; + color: #E9E9E9; +} +.benchmark-banner p a { + color: #B4F0FF; +} \ No newline at end of file diff --git a/docs/sphinx_setup/_static/download/OV_2022_models_supported.pdf b/docs/sphinx_setup/_static/download/OV_2022_models_supported.pdf deleted file mode 100644 index f1b360daa88..00000000000 Binary files a/docs/sphinx_setup/_static/download/OV_2022_models_supported.pdf and /dev/null differ diff --git a/docs/sphinx_setup/_static/download/OV_2023_models_supported.pdf b/docs/sphinx_setup/_static/download/OV_2023_models_supported.pdf deleted file mode 100644 index 0d13ba1bf64..00000000000 Binary files a/docs/sphinx_setup/_static/download/OV_2023_models_supported.pdf and /dev/null differ