natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS Banchmark full update to master from 24 1 port (#24313) 

this port to master for:
https://github.com/openvinotoolkit/openvino/pull/24144
https://github.com/openvinotoolkit/openvino/pull/24230
https://github.com/openvinotoolkit/openvino/pull/24256

---------

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
This commit is contained in:
Andrzej Kopytko 2024-04-30 09:49:31 +02:00 committed by GitHub
parent f09d1388f0
commit 25f211ee75
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 1341 additions and 751 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

135
docs/articles_en/about-openvino/performance-benchmarks.rst
View File

@ -17,10 +17,12 @@ Performance Benchmarks
performance-benchmarks/getting-performance-numbers
This page presents benchmark results for `Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
and :doc:`OpenVINO Model Server <../ovms_what_is_openvino_model_server>`, for a representative selection of public neural networks and Intel® devices.
The results may help you decide which hardware to use in your applications or plan AI workload for the hardware you have already implemented in your solutions.
Click the buttons below to see the chosen benchmark data.
This page presents benchmark results for
`Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
and :doc:`OpenVINO Model Server <../ovms_what_is_openvino_model_server>`, for a representative
selection of public neural networks and Intel® devices. The results may help you decide which
hardware to use in your applications or plan AI workload for the hardware you have already
implemented in your solutions. Click the buttons below to see the chosen benchmark data.
.. grid:: 1 1 2 2
:gutter: 4
@ -66,36 +68,41 @@ Please visit the tabs below for more information on key performance indicators a
throughput per dollar of system cost is the best measure of value. The value KPI is
calculated as “Throughput measured as inferences per second / price of inference engine”.
This means for a 2 socket system 2x the price of a CPU is used. Prices are as per
date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
date of benchmarking and sources can be found as links in the Hardware Platforms (PDF)
description below.
.. tab-item:: Efficiency
:sync: efficiency
System power is a key consideration from the edge to the data center. When selecting
deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider.
Intel designs provide excellent power efficiency for running deep learning workloads.
The efficiency KPI is calculated as “Throughput measured as inferences per second / TDP of
inference engine”. This means for a 2 socket system 2x the power dissipation (TDP) of a CPU is used.
TDP-values are as per date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
deep learning solutions, power efficiency (throughput/watt) is a critical factor to
consider. Intel designs provide excellent power efficiency for running deep learning
workloads. The efficiency KPI is calculated as “Throughput measured as inferences per
second / TDP of inference engine”. This means for a 2 socket system 2x the power
dissipation (TDP) of a CPU is used. TDP-values are as per date of benchmarking and sources
can be found as links in the Hardware Platforms (PDF) description below.
.. tab-item:: Latency
:sync: latency
This measures the synchronous execution of inference requests and is reported in milliseconds.
Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before
the next is started. This performance metric is relevant in usage scenarios where a single image
input needs to be acted upon as soon as possible. An example would be the healthcare sector where
medical personnel only request analysis of a single ultra sound scanning image or in real-time or
near real-time applications for example an industrial robot's response to actions in its environment
or obstacle avoidance for autonomous vehicles.
This measures the synchronous execution of inference requests and is reported in
milliseconds. Each inference request (for example: preprocess, infer, postprocess) is
allowed to complete before the next is started. This performance metric is relevant in
usage scenarios where a single image input needs to be acted upon as soon as possible. An
example would be the healthcare sector where medical personnel only request analysis of a
single ultra sound scanning image or in real-time or near real-time applications for
example an industrial robot's response to actions in its environment or obstacle avoidance
for autonomous vehicles.
.. tab-item:: Workload Parameters
:sync: workloadparameters
The workload parameters affect the performance results of the different models we use for benchmarking.
Image processing models have different image size definitions and the Natural Language Processing models
have different max token list lengths. All these can be found in detail in the :doc:`FAQ section <performance-benchmarks/performance-benchmarks-faq>`.
All models are executed using a batch size of 1. Below are the parameters for the GenAI models we display.
The workload parameters affect the performance results of the different models we use for
benchmarking. Image processing models have different image size definitions and the
Natural Language Processing models have different max token list lengths. All these can
be found in detail in the :doc:`FAQ section <performance-benchmarks/performance-benchmarks-faq>`.
All models are executed using a batch size of 1. Below are the parameters for the GenAI
models we display.
* Input tokens: 1024,
* Output tokens: 128,
@ -119,21 +126,21 @@ For a listing of all platforms and configurations used for testing, refer to the
.. grid-item::
.. button-link:: ../_static/benchmarks_files/OV-2024.0-platform_list.pdf
.. button-link:: ../_static/benchmarks_files/OV-2024.1-platform_list.pdf
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Hardware Platforms [PDF]
.. button-link:: ../_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.1-system-info-detailed.xlsx
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. button-link:: ../_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.1-Performance-Data.xlsx
:color: primary
:outline:
:expand:
@ -141,15 +148,16 @@ For a listing of all platforms and configurations used for testing, refer to the
:material-regular:`download;1.5em` Click for Performance Data [XLSX]
The OpenVINO benchmark setup includes a single system with OpenVINO™, as well as the benchmark application installed.
It measures the time spent on actual inference (excluding any pre or post processing) and then reports on the inferences
per second (or Frames Per Second).
The OpenVINO benchmark setup includes a single system with OpenVINO™, as well as the benchmark
application installed. It measures the time spent on actual inference (excluding any pre or post
processing) and then reports on the inferences per second (or Frames Per Second).
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime libraries and exposes a set of
models via a convenient inference API over gRPC or HTTP/REST. Its benchmark results are measured with the configuration of
multiple-clients-single-server, using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The connection is dedicated
only to measuring performance.
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime
libraries and exposes a set of models via a convenient inference API over gRPC or HTTP/REST.
Its benchmark results are measured with the configuration of multiple-clients-single-server,
using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The
connection is dedicated only to measuring performance.
.. dropdown:: See more details about OVMS benchmark setup
@ -158,22 +166,28 @@ only to measuring performance.
.. image:: ../_static/images/performance_benchmarks_ovms_02.png
:alt: OVMS Benchmark Setup Diagram
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on)
requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application
in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into
the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it
listens to (and answers) requests from clients. OpenVINO™ Model Server is run on the same
system as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models
served by OpenVINO™ Model Server are located in a local file system mounted into the docker
container. The OpenVINO™ Model Server instance communicates with other components via ports
over a dedicated docker network.
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3
programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™
Model Server before it will send a new next request. The role played by the clients is also verification of responses.
* **Clients** are run in separated physical machine referred to as client platform. Clients
are implemented in Python3 programming language based on TensorFlow* API and they work as
parallel processes. Each client waits for a response from OpenVINO™ Model Server before it
will send a new next request. The role played by the clients is also verification of
responses.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is
counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by
Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes
impact of physical network on reported metrics.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for
this purpose. Its main role is counting of requests forwarded from clients to OpenVINO™
Model Server, estimating its latency, and sharing this information by Prometheus service.
The reason of locating the load balancer on the client site is to simulate real life
scenario that includes impact of physical network on reported metrics.
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process,
downloading metrics from the load balancer, and presenting the final report of the execution.
* **Execution Controller** is launched on the client platform. It is responsible for
synchronization of the whole measurement process, downloading metrics from the load
balancer, and presenting the final report of the execution.
@ -181,12 +195,16 @@ only to measuring performance.
<h2>Test performance yourself</h2>
You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers <performance-benchmarks/getting-performance-numbers>`.
You can also test performance for your system yourself, following the guide on
:doc:`getting performance numbers <performance-benchmarks/getting-performance-numbers>`.
Performance of a particular application can also be evaluated virtually using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit.
To learn more about it, visit `the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
Performance of a particular application can also be evaluated virtually using
`Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions
of the Intel® Distribution of the OpenVINO™ Toolkit. To learn more about it, visit
`the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or
`create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
.. raw:: html
@ -194,17 +212,22 @@ or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devc
<h2>Disclaimers</h2>
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2024.0, as of March 06, 2024.
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release
2024.1, as of April 17, 2024.
* OpenVINO Model Server performance results are based on release 2024.0, as of March 15, 2024.
* OpenVINO Model Server performance results are based on release
2024.0, as of March 15, 2024.
The results may not reflect all publicly available updates. Intel technologies' features and benefits depend on system configuration
and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
The results may not reflect all publicly available updates. Intel technologies' features and
benefits depend on system configuration and may require enabled hardware, software, or service
activation. Learn more at intel.com, or from the OEM or retailer.
See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at `www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
Performance varies by use, configuration and other factors. Learn more at
`www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
Your costs and results may vary.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree
for non-Intel products.

42
docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst
View File

@ -29,16 +29,16 @@ the table for more information.
* - bert-base-cased
- SST-2_bert_cased_padded
- spearman@cosine
- 3.28%
- 2.93%
- 2.68%
- 2.91%
- 2.76%
- 2.72%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
- 0.07%
- 0.19%
- -0.03%
- 0.13%
- 0.03%
- 0.11%
* - efficientdet-d0
- COCO2017_detection_91cl
@ -57,7 +57,7 @@ the table for more information.
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
- %
-
- -0.97%
- -0.97%
- -0.95%
@ -85,9 +85,9 @@ the table for more information.
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
- -6.32%
- 6.40%
- -6.28%
- 6.41%
- 6.46%
- 6.40%
* - yolo_v3_tiny
- COCO2017_detection_80cl
@ -131,6 +131,13 @@ the table for more information.
- 3.49
- 3.19
-
* - Falcon-7b-instruct
- Bai Ze (65%), GPT4All (25%), GPTeacher (5%), RefinedWeb-english (5%)
- ppl
-
-
-
-
.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
:header-rows: 1
@ -157,7 +164,7 @@ the table for more information.
- 0.04%
- 0.04%
- 0.04%
- 0.05%
- 0.06%
- 0.05%
* - efficientdet-d0
- COCO2017_detection_91cl
@ -172,9 +179,9 @@ the table for more information.
- coco_orig_precision
- -0.01%
- -0.02%
- %
- -0.01%
- 0.09%
- 0.02%
- 0.00%
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
@ -206,7 +213,7 @@ the table for more information.
- 0.01%
- 0.01%
- 0.04%
- -0.04%
- -0.02%
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
@ -251,8 +258,8 @@ the table for more information.
- LIAON-5B
- CLIP
-
-
-
- 31.3
- 22.4
-
-
* - Mistral-7b
@ -263,7 +270,14 @@ the table for more information.
- 3.19
-
-
* - Falcon-7b-instruct
- Bai Ze (65%), GPT4All (25%), GPTeacher (5%), RefinedWeb-english (5%)
- ppl
-
-
-
-
-
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured

10
docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst
View File

@ -32,7 +32,6 @@ Performance Information F.A.Q.
All of the performance benchmarks are generated using the
open-source tool within the Intel® Distribution of OpenVINO™ toolkit
called :doc:`benchmark_app <../../learn-openvino/openvino-samples/benchmark-tool>`.
This tool is available for Python and C++ apps.
For a simple instruction on testing performance, see the :doc:`Getting Performance Numbers Guide <getting-performance-numbers>`.
@ -53,6 +52,10 @@ Performance Information F.A.Q.
- THUDM
- Transformer
- 32K
* - `Falcon-7b-instruct <https://huggingface.co/tiiuae/falcon-7b-instruct>`__
- Hugginface
- Causal Decoder-only
- 2048
* - `Llama-2-7b-chat <https://ai.meta.com/llama/>`__
- Meta AI
- Auto regressive language
@ -115,8 +118,9 @@ Performance Information F.A.Q.
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
Before purchasing any hardware, you can test and run models remotely, using
`Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
For more details, see the :doc:`Supported Devices <../compatibility-and-support/supported-devices>`.
documentation. Before purchasing any hardware, you can test and run
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
.. dropdown:: How can I optimize my models for better performance or accuracy?

700
docs/articles_en/about-openvino/release-notes-openvino.rst
View File

@ -1,4 +1,6 @@
.. {#openvino_release_notes}
.. meta::
:description: See what has changed in OpenVINO with the latest release, as well as all
previous releases in this year's cycle.
OpenVINO Release Notes
=============================
@ -11,48 +13,45 @@ OpenVINO Release Notes
release-notes-openvino/release-policy
2024.0 - 06 March 2024
2024.1 - 24 April 2024
#############################
:doc:`System Requirements <./release-notes-openvino/system-requirements>` | :doc:`Installation Guides <./../get-started/install-openvino>`
:doc:`System Requirements <./release-notes-openvino/system-requirements>` | :doc:`Release policy <./release-notes-openvino/release-policy>` | :doc:`Installation Guides <./../get-started/install-openvino>`
What's new
+++++++++++++++++++++++++++++
* More Generative AI coverage and framework integrations to minimize code changes.
* More Gen AI coverage and framework integrations to minimize code changes.
* Improved out-of-the-box experience for TensorFlow sentence encoding models through the
installation of OpenVINO™ toolkit Tokenizers.
* New and noteworthy models validated:
Mistral, StableLM-tuned-alpha-3b, and StableLM-Epoch-3B.
* OpenVINO™ toolkit now supports Mixture of Experts (MoE), a new architecture that helps
process more efficient generative models through the pipeline.
* JavaScript developers now have seamless access to OpenVINO API. This new binding enables a
smooth integration with JavaScript API.
* Mixtral and URLNet models optimized for performance improvements on Intel® Xeon® processors.
* Stable Diffusion 1.5, ChatGLM3-6B, and Qwen-7B models optimized for improved inference speed
on Intel® Core™ Ultra processors with integrated GPU.
* Support for Falcon-7B-Instruct, a GenAI Large Language Model (LLM) ready-to-use chat/instruct
model with superior performance metrics.
* New Jupyter Notebooks added: YOLO V9, YOLO V8 Oriented Bounding Boxes Detection (OOB), Stable
Diffusion in Keras, MobileCLIP, RMBG-v1.4 Background Removal, Magika, TripoSR, AnimateAnyone,
LLaVA-Next, and RAG system with OpenVINO and LangChain.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Broader LLM model support and more model compression techniques.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Improved quality on INT4 weight compression for LLMs by adding the popular technique,
Activation-aware Weight Quantization, to the Neural Network Compression Framework (NNCF).
This addition reduces memory requirements and helps speed up token generation.
* Experience enhanced LLM performance on Intel® CPUs, with internal memory state enhancement,
and INT8 precision for KV-cache. Specifically tailored for multi-query LLMs like ChatGLM.
* The OpenVINO™ 2024.0 release makes it easier for developers, by integrating more OpenVINO™
features with the Hugging Face ecosystem. Store quantization configurations for popular
models directly in Hugging Face to compress models into INT4 format while preserving
accuracy and performance.
* LLM compilation time reduced through additional optimizations with compressed embedding.
Improved 1st token performance of LLMs on 4th and 5th generations of Intel® Xeon® processors
with Intel® Advanced Matrix Extensions (Intel® AMX).
* Better LLM compression and improved performance with oneDNN, INT4, and INT8 support for
Intel® Arc™ GPUs.
* Significant memory reduction for select smaller GenAI models on Intel® Core™ Ultra processors
with integrated GPU.
* More portability and performance to run AI at the edge, in the cloud, or locally.
* A preview plugin architecture of the integrated Neural Processor Unit (NPU) as part of
Intel® Core™ Ultra processor (codename Meteor Lake) is now included in the main OpenVINO™
package on PyPI.
* Improved performance on ARM by enabling the ARM threading library. In addition, we now
support multi-core ARM platforms and enabled FP16 precision by default on MacOS.
* New and improved LLM serving samples from OpenVINO Model Server for multi-batch inputs and
Retrieval Augmented Generation (RAG).
* The preview NPU plugin for Intel® Core™ Ultra processors is now available in the OpenVINO
open-source GitHub repository, in addition to the main OpenVINO package on PyPI.
* The JavaScript API is now more easily accessible through the npm repository, enabling
JavaScript developers' seamless access to the OpenVINO API.
* FP16 inference on ARM processors now enabled for the Convolutional Neural Network (CNN) by
default.
OpenVINO™ Runtime
@ -61,218 +60,467 @@ OpenVINO™ Runtime
Common
-----------------------------
* The legacy API for CPP and Python bindings has been removed.
* StringTensor support has been extended by operators such as ``Gather``, ``Reshape``, and
``Concat``, as a foundation to improve support for tokenizer operators and compliance with
the TensorFlow Hub.
* oneDNN has been updated to v3.3.
(`see oneDNN release notes <https://github.com/oneapi-src/oneDNN/releases>`__).
* Unicode file paths for cached models are now supported on Windows.
* Pad pre-processing API to extend input tensor on edges with constants.
* A fix for inference failures of certain image generation models has been implemented
(fused I/O port names after transformation).
* Compiler's warnings-as-errors option is now on, improving the coding criteria and quality.
Build warnings will not be allowed for new OpenVINO code and the existing warnings have been
fixed.
AUTO Inference Mode
-----------------------------
* Returning the ov::enable_profiling value from ov::CompiledModel is now supported.
CPU Device Plugin
-----------------------------
* LLM performance on Intel® CPU platforms has been improved for systems based on AVX2 and
AVX512, using dynamic quantization and internal memory state optimization, such as INT8
precision for KV-cache. 13th and 14th generations of Intel® Core™ processors and Intel® Core™
Ultra processors use AVX2 for CPU execution, and these platforms will benefit from speedup.
Enable these features by setting ``"DYNAMIC_QUANTIZATION_GROUP_SIZE":"32"`` and
``"KV_CACHE_PRECISION":"u8"`` in the configuration file.
* The ``ov::affinity`` API configuration is now deprecated and will be removed in release
2025.0.
* The following have been improved and optimized:
* Multi-query structure LLMs (such as ChatGLM 2/3) for BF16 on the 4th and 5th generation
Intel® Xeon® Scalable processors.
* `Mixtral <https://huggingface.co/docs/transformers/model_doc/mixtral>`__ model performance.
* 8-bit compressed LLM compilation time and memory usage, valuable for models with large
embeddings like `Qwen <https://github.com/QwenLM/Qwen>`__.
* Convolutional networks in FP16 precision on ARM platforms.
* 1st token performance of LLMs has been improved on the 4th and 5th generations of Intel® Xeon®
processors with Intel® Advanced Matrix Extensions (Intel® AMX).
* LLM compilation time and memory footprint have been improved through additional optimizations
with compressed embeddings.
* Performance of MoE (e.g. Mixtral), Gemma, and GPT-J has been improved further.
* Performance has been improved significantly for a wide set of models on ARM devices.
* FP16 inference precision is now the default for all types of models on ARM devices.
* CPU architecture-agnostic build has been implemented, to enable unified binary distribution
on different ARM devices.
GPU Device Plugin
-----------------------------
* The following have been improved and optimized:
* LLM first token latency has been improved on both integrated and discrete GPU platforms.
* For the ChatGLM3-6B model, average token latency has been improved on integrated GPU platforms.
* For Stable Diffusion 1.5 FP16 precision, performance has been improved on Intel® Core™ Ultra
processors.
* Average token latency for LLMs on integrated GPU (iGPU) platforms, using INT4-compressed
models with large context size on Intel® Core™ Ultra processors.
* LLM beam search performance on iGPU. Both average and first-token latency decrease may be
expected for larger context sizes.
* Multi-batch performance of YOLOv5 on iGPU platforms.
NPU Device Plugin
-----------------------------
* Memory usage for LLMs has been optimized, enabling '7B' models with larger context on
16Gb platforms.
NPU Device Plugin (preview feature)
-----------------------------------
* The NPU plugin for OpenVINO™ is now available through PyPI (run “pip install openvino”).
* NPU Plugin is now part of the OpenVINO GitHub repository. All the most recent plugin changes
will be immediately available in the repo. Note that NPU is part of Intel® Core™ Ultra
processors.
* New OpenVINO™ notebook “Hello, NPU!” introducing NPU usage with OpenVINO has been added.
* Version 22H2 or later is required for Microsoft Windows® 11 64-bit to run inference on NPU.
OpenVINO Python API
-----------------------------
* ``.add_extension`` method signatures have been aligned, improving API behavior for better
user experience.
* GIL-free creation of RemoteTensors is now used - holding GIL means that the process is not suited
for multithreading and removing the GIL lock will increase performance which is critical for
the concept of Remote Tensors.
* Packed data type BF16 on the Python API level has been added, opening a new way of supporting
data types not handled by numpy.
* 'pad' operator support for ov::preprocess::PrePostProcessorItem has been added.
* ov.PartialShape.dynamic(int) definition has been provided.
OpenVINO C API
-----------------------------
* ov_property_key_cache_mode (C++ ov::cache_mode) now enables the ``optimize_size`` and
``optimize_speed`` modes to set/get model cache.
* The VA surface on Windows exception has been fixed.
* Two new pre-processing APIs for scale and mean have been added.
OpenVINO Node.js API
-----------------------------
* OpenVINO - `JS bindings <https://docs.openvino.ai/2024/api/nodejs_api/nodejs_api.html>`__
are consistent with the OpenVINO C++ API.
* A new distribution channel is now available: Node Package Manager (npm) software registry
(:doc:`check the installation guide <../get-started/install-openvino/install-openvino-npm>`).
* JavaScript API is now available for Windows users, as some limitations for platforms other
than Linux have been removed.
* New methods to align JavaScript API with CPP API have been added, such as
CompiledModel.exportModel(), core.import_model(), Core set/get property and Tensor.get_size(),
and Model.is_dynamic().
* Documentation has been extended to help developers start integrating JavaScript applications
with OpenVINO™.
TensorFlow Framework Support
-----------------------------
* String tensors are now natively supported, handled on input, output, and intermediate layers
(`PR #22024 <https://github.com/openvinotoolkit/openvino/pull/22024>`__).
* `tf.keras.layers.TextVectorization tokenizer <https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization>`__
is now supported.
* Conversion of models with Variable and HashTable (dictionary) resources has been improved.
* 8 NEW operations have been added
(`see the list here, marked as NEW <https://github.com/openvinotoolkit/openvino/blob/releases/2024/1/src/frontends/tensorflow/docs/supported_ops.md>`__).
* 10 operations have received complex tensor support.
* Input tensor names for TF1 models have been adjusted to have a single name per input.
* Hugging Face model support coverage has increased significantly, due to:
* TensorFlow Hub universal-sentence-encoder-multilingual inferred out of the box
* string tensors supported for ``Gather``, ``Concat``, and ``Reshape`` operations
* integration with openvino-tokenizers module - importing openvino-tokenizers automatically
patches TensorFlow FE with the required translators for models with tokenization
* extraction of input signature of a model in memory has been fixed,
* reading of variable values for a model in memory has been fixed.
* Fallback for Model Optimizer by operation to the legacy Frontend is no longer available.
Fallback by .json config will remain until Model Optimizer is discontinued
(`PR #21523 <https://github.com/openvinotoolkit/openvino/pull/21523>`__).
* Support for the following has been added:
* Mutable variables and resources such as HashTable*, Variable, VariableV2
(`PR #22270 <https://github.com/openvinotoolkit/openvino/pull/22270>`__).
* New tensor types: tf.u16, tf.u32, and tf.u64
(`PR #21864 <https://github.com/openvinotoolkit/openvino/pull/21864>`__).
* 14 NEW Ops*.
`Check the list here (marked as NEW) <https://github.com/openvinotoolkit/openvino/blob/releases/2024/0/src/frontends/tensorflow/docs/supported_ops.md>`__.
* TensorFlow 2.15
(`PR #22180 <https://github.com/openvinotoolkit/openvino/pull/22180>`__).
* The following issues have been fixed:
* UpSampling2D conversion crashed when input type as int16
(`PR #20838 <https://github.com/openvinotoolkit/openvino/pull/20838>`__).
* IndexError list index for Squeeze
(`PR #22326 <https://github.com/openvinotoolkit/openvino/pull/22326>`__).
* Correct FloorDiv computation for signed integers
(`PR #22684 <https://github.com/openvinotoolkit/openvino/pull/22684>`__).
* Fixed bad cast error for tf.TensorShape to ov.PartialShape
(`PR #22813 <https://github.com/openvinotoolkit/openvino/pull/22813>`__).
* Fixed reading tf.string attributes for models in memory
(`PR #22752 <https://github.com/openvinotoolkit/openvino/pull/22752>`__).
ONNX Framework Support
-----------------------------
* ONNX Frontend now uses the OpenVINO API 2.0.
PyTorch Framework Support
-----------------------------
* Names for outputs unpacked from dict or tuple are now clearer
(`PR #22821 <https://github.com/openvinotoolkit/openvino/pull/22821>`__).
* FX Graph (torch.compile) now supports kwarg inputs, improving data type coverage.
(`PR #22397 <https://github.com/openvinotoolkit/openvino/pull/22397>`__).
* ModuleExtension, a new type of extension for PyTorch models is now supported
(`PR #23536 <https://github.com/openvinotoolkit/openvino/pull/23536>`__).
* 22 NEW operations have been added.
* Experimental support for models produced by torch.export (FX graph) has been added
(`PR #23815 <https://github.com/openvinotoolkit/openvino/pull/23815>`__).
ONNX Framework Support
-----------------------------
* 8 new operations have been added.
OpenVINO Model Server
+++++++++++++++++++++++++++++
* OpenVINO™ Runtime backend used is now 2024.0.
* Text generation demo now supports multi batch size, with streaming and unary clients.
* The REST client now supports servables based on mediapipe graphs, including python pipeline
nodes.
* Included dependencies have received security-related updates.
* Reshaping a model in runtime based on the incoming requests (auto shape and auto batch size)
is deprecated and will be removed in the future. Using OpenVINO's dynamic shape models is
recommended instead.
* OpenVINO™ Runtime backend used is now 2024.1
* OpenVINO™ models with String data type on output are supported. Now, OpenVINO™ Model Server
can support models with input and output of the String type, so developers can take advantage
of the tokenization built into the model as the first layer. Developers can also rely on any
postprocessing embedded into the model which returns text only. Check the
`demo on string input data with the universal-sentence-encoder model <https://docs.openvino.ai/2024/ovms_demo_universal-sentence-encoder.html>`__
and the
`String output model demo <https://github.com/openvinotoolkit/model_server/tree/main/demos/image_classification_with_string_output>`__.
* MediaPipe Python calculators have been updated to support relative paths for all related
configuration and Python code files. Now, the complete graph configuration folder can be
deployed in an arbitrary path without any code changes.
* KServe REST API support has been extended to properly handle the string format in JSON body,
just like the binary format compatible with NVIDIA Triton™.
* `A demo showcasing a full RAG algorithm <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/rag_chatbot>`__
fully delegated to the model server has been added.
Neural Network Compression Framework (NNCF)
+++++++++++++++++++++++++++++++++++++++++++
Neural Network Compression Framework
++++++++++++++++++++++++++++++++++++++++++
* The `Activation-aware Weight Quantization (AWQ) <https://arxiv.org/abs/2306.00978>`__
algorithm for data-aware 4-bit weights compression is now available. It facilitates better
accuracy for compressed LLMs with high ratio of 4-bit weights. To enable it, use the
dedicated ``awq`` optional parameter of ``the nncf.compress_weights()`` API.
* ONNX models are now supported in Post-training Quantization with Accuracy Control, through
the ``nncf.quantize_with_accuracy_control()``, method. It may be used for models in the
OpenVINO IR and ONNX formats.
* A `weight compression example tutorial <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__
is now available, demonstrating how to find the appropriate hyperparameters for the TinyLLama
model from the Hugging Face Transformers, as well as other LLMs, with some modifications.
* Model subgraphs can now be defined in the ignored scope for INT8 Post-training Quantization,
nncf.quantize(), which simplifies excluding accuracy-sensitive layers from quantization.
* A batch size of more than 1 is now partially supported for INT8 Post-training Quantization,
speeding up the process. Note that it is not recommended for transformer-based models as it
may impact accuracy. Here is an
`example demo <https://github.com/openvinotoolkit/nncf/blob/develop/examples/quantization_aware_training/torch/resnet18/README.md>`__.
* Now it is possible to apply fine-tuning on INT8 models after Post-training Quantization to
improve model accuracy and make it easier to move from post-training to training-aware
quantization. Here is an
`example demo <https://github.com/openvinotoolkit/nncf/blob/develop/examples/quantization_aware_training/torch/resnet18/README.md>`__.
OpenVINO Tokenizers
++++++++++++++++++++++++++++++++++++++++++
OpenVINO Tokenizer
+++++++++++++++++++++++++++++
* TensorFlow support has been extended - TextVectorization layer translation:
* Regex support has been improved.
* Model coverage has been improved.
* Tokenizer metadata has been added to rt_info.
* Limited support for Tensorflow Text models has been added: convert MUSE for TF Hub with
string inputs.
* OpenVINO Tokenizers have their own repository now:
`/openvino_tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
* Aligned existing ops with TF ops and added a translator for them.
* Added new ragged tensor ops and string ops.
* A new tokenizer type, RWKV is now supported:
* Added Trie tokenizer and Fuse op for ragged tensors.
* A new way to get OV Tokenizers: build a vocab from file.
* Tokenizer caching has been redesigned to work with the OpenVINO™ model caching mechanism.
Other Changes and Known Issues
+++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
Jupyter Notebooks
-----------------------------
The default branch for the OpenVINO™ Notebooks repository has been changed from 'main' to
'latest'. The 'main' branch of the notebooks repository is now deprecated and will be maintained
until September 30, 2024.
The new branch, 'latest', offers a better user experience and simplifies maintenance due to
significant refactoring and an improved directory naming structure.
Use the local
`README.md <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/README.md>`__
file and OpenVINO™ Notebooks at
`GitHub Pages <https://openvinotoolkit.github.io/openvino_notebooks/>`__
to navigate through the content.
The following notebooks have been updated or newly added:
* `Mobile language assistant with MobileVLM <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/mobilevlm-language-assistant>`__
* `Depth estimation with DepthAnything <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/depth-anything>`__
* `Kosmos-2 <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/kosmos2-multimodal-large-language-model>`__
* `Zero-shot Image Classification with SigLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/siglip-zero-shot-image-classification>`__
* `Personalized image generation with PhotoMaker <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/photo-maker>`__
* `Voice tone cloning with OpenVoice <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvoice>`__
* `Line-level text detection with Surya <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/surya-line-level-text-detection>`__
* `InstantID: Zero-shot Identity-Preserving Generation using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/instant-id>`__
* `Tutorial for Big Image Transfer (BIT) model quantization using NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/big-transfer-quantization>`__
* `Tutorial for OpenVINO Tokenizers integration into inference pipelines <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvino-tokenizers>`__
* `LLM chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot.ipynb>`__ and
`LLM RAG pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/rag-chatbot.ipynb>`__
have received integration with new models: minicpm-2b-dpo, gemma-7b-it, qwen1.5-7b-chat, baichuan2-7b-chat
* `Grounded Segment Anything <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/grounded-segment-anything/grounded-segment-anything.ipynb>`__
* `Visual Content Search with MobileCLIP <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/mobileclip-video-search/mobileclip-video-search.ipynb>`__
* `YOLO V8 Oriented Bounding Box Detection Optimization <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/yolov8-optimization/yolov8-obb.ipynb>`__
* `Magika: AI-powered fast and efficient file type identification <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/magika-content-type-recognition/magika-content-type-recognition.ipynb>`__
* `Keras Stable Diffusion <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-keras-cv/stable-diffusion-keras-cv.ipynb>`__
* `RMBG background removal <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/rmbg-background-removal/rmbg-background-removal.ipynb>`__
* `AnimateAnyone: pose guided image to video generation <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/animate-anyone/animate-anyone.ipynb>`__
* `LLaVA-Next visual-language assistant <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llava-next-multimodal-chatbot/llava-next-multimodal-chatbot.ipynb>`__
* `TripoSR: single image 3d reconstruction <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/triposr-3d-reconstruction/triposr-3d-reconstruction.ipynb>`__
* `RAG system with OpenVINO and LangChain <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain.ipynb>`__
Known issues
Known Issues
-----------------------------
| **Component - CPU Plugin**
| *ID* - N/A
| *Description:*
| Starting with 24.0, model inputs and outputs will no longer have tensor names, unless
explicitly set to align with the PyTorch framework behavior.
| Default CPU pinning policy on Windows has been changed to follow Windows' policy
instead of controlling the CPU pinning in the OpenVINO plugin. This brings certain dynamic or
performance variance on Windows. Developers can use ov::hint::enable_cpu_pinning to enable
or disable CPU pinning explicitly.
| **Component - GPU runtime**
| *ID* - 132376
| *Description:*
| First-inference latency slow down for LLMs on Intel® Core™ Ultra processors. Up to 10-20%
drop may occur due to radical memory optimization for processing long sequences
(about 1.5-2 GB reduced memory usage).
| **Component - CPU runtime**
| **Component - Hardware Configuration**
| *ID* - N/A
| *Description:*
| Performance results (first token latency) may vary from those offered by the previous OpenVINO version, for
“latency” hint inference of LLMs with long prompts on Xeon platforms with 2 or more
sockets. The reason is that all CPU cores of just the single socket running the application
are employed, lowering the memory overhead for LLMs when numa control is not used.
| *Workaround:*
| The behavior is expected but stream and thread configuration may be used to include cores
from all sockets.
| Reduced performance for LLMs may be observed on newer CPUs. To mitigate, modify the default settings in BIOS to
| change the system into 2 NUMA node system:
| 1. Enter the BIOS configuration menu.
| 2. Select EDKII Menu -> Socket Configuration -> Uncore Configuration -> Uncore General Configuration -> SNC.
| 3. The SNC setting is set to *AUTO* by default. Change the SNC setting to *disabled* to configure one NUMA node per processor socket upon boot.
| 4. After system reboot, confirm the NUMA node setting using: `numatcl -H`. Expect to see only nodes 0 and 1 on a
| 2-socket system with the following mapping:
| Node - 0 - 1
| 0 - 10 - 21
| 1 - 21 - 10
Previous 2024 releases
+++++++++++++++++++++++++++++
.. dropdown:: 2024.0 - 06 March 2024
:animate: fade-in-slide-down
:color: secondary
**What's new**
* More Generative AI coverage and framework integrations to minimize code changes.
* Improved out-of-the-box experience for TensorFlow sentence encoding models through the
installation of OpenVINO™ toolkit Tokenizers.
* New and noteworthy models validated:
Mistral, StableLM-tuned-alpha-3b, and StableLM-Epoch-3B.
* OpenVINO™ toolkit now supports Mixture of Experts (MoE), a new architecture that helps
process more efficient generative models through the pipeline.
* JavaScript developers now have seamless access to OpenVINO API. This new binding enables a
smooth integration with JavaScript API.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Improved quality on INT4 weight compression for LLMs by adding the popular technique,
Activation-aware Weight Quantization, to the Neural Network Compression Framework (NNCF).
This addition reduces memory requirements and helps speed up token generation.
* Experience enhanced LLM performance on Intel® CPUs, with internal memory state enhancement,
and INT8 precision for KV-cache. Specifically tailored for multi-query LLMs like ChatGLM.
* The OpenVINO™ 2024.0 release makes it easier for developers, by integrating more OpenVINO™
features with the Hugging Face ecosystem. Store quantization configurations for popular
models directly in Hugging Face to compress models into INT4 format while preserving
accuracy and performance.
* More portability and performance to run AI at the edge, in the cloud, or locally.
* A preview plugin architecture of the integrated Neural Processor Unit (NPU) as part of
Intel® Core™ Ultra processor (codename Meteor Lake) is now included in the main OpenVINO™
package on PyPI.
* Improved performance on ARM by enabling the ARM threading library. In addition, we now
support multi-core ARM processors and enabled FP16 precision by default on MacOS.
* New and improved LLM serving samples from OpenVINO Model Server for multi-batch inputs and
Retrieval Augmented Generation (RAG).
**OpenVINO™ Runtime**
*Common*
* The legacy API for CPP and Python bindings has been removed.
* StringTensor support has been extended by operators such as ``Gather``, ``Reshape``, and
``Concat``, as a foundation to improve support for tokenizer operators and compliance with
the TensorFlow Hub.
* oneDNN has been updated to v3.3.
(`see oneDNN release notes <https://github.com/oneapi-src/oneDNN/releases>`__).
*CPU Device Plugin*
* LLM performance on Intel® CPU platforms has been improved for systems based on AVX2 and
AVX512, using dynamic quantization and internal memory state optimization, such as INT8
precision for KV-cache. 13th and 14th generations of Intel® Core™ processors and Intel® Core™
Ultra processors use AVX2 for CPU execution, and these platforms will benefit from speedup.
Enable these features by setting ``"DYNAMIC_QUANTIZATION_GROUP_SIZE":"32"`` and
``"KV_CACHE_PRECISION":"u8"`` in the configuration file.
* The ``ov::affinity`` API configuration is now deprecated and will be removed in release
2025.0.
* The following have been improved and optimized:
* Multi-query structure LLMs (such as ChatGLM 2/3) for BF16 on the 4th and 5th generation
Intel® Xeon® Scalable processors.
* `Mixtral <https://huggingface.co/docs/transformers/model_doc/mixtral>`__ model performance.
* 8-bit compressed LLM compilation time and memory usage, valuable for models with large
embeddings like `Qwen <https://github.com/QwenLM/Qwen>`__.
* Convolutional networks in FP16 precision on ARM processors.
*GPU Device Plugin*
* The following have been improved and optimized:
* Average token latency for LLMs on integrated GPU (iGPU) platforms, using INT4-compressed
models with large context size on Intel® Core™ Ultra processors.
* LLM beam search performance on iGPU. Both average and first-token latency decrease may be
expected for larger context sizes.
* Multi-batch performance of YOLOv5 on iGPU platforms.
* Memory usage for LLMs has been optimized, enabling '7B' models with larger context on
16Gb platforms.
*NPU Device Plugin (preview feature)*
* The NPU plugin for OpenVINO™ is now available through PyPI (run “pip install openvino”).
*OpenVINO Python API*
* ``.add_extension`` method signatures have been aligned, improving API behavior for better
user experience.
*OpenVINO C API*
* ov_property_key_cache_mode (C++ ov::cache_mode) now enables the ``optimize_size`` and
``optimize_speed`` modes to set/get model cache.
* The VA surface on Windows exception has been fixed.
*OpenVINO Node.js API*
* OpenVINO - `JS bindings <https://docs.openvino.ai/2024/api/nodejs_api/nodejs_api.html>`__
are consistent with the OpenVINO C++ API.
* A new distribution channel is now available: Node Package Manager (npm) software registry
(:doc:`check the installation guide <../get-started/install-openvino/install-openvino-npm>`).
* JavaScript API is now available for Windows users, as some limitations for platforms other
than Linux have been removed.
*TensorFlow Framework Support*
* String tensors are now natively supported, handled on input, output, and intermediate layers
(`PR #22024 <https://github.com/openvinotoolkit/openvino/pull/22024>`__).
* TensorFlow Hub universal-sentence-encoder-multilingual inferred out of the box
* string tensors supported for ``Gather``, ``Concat``, and ``Reshape`` operations
* integration with openvino-tokenizers module - importing openvino-tokenizers automatically
patches TensorFlow FE with the required translators for models with tokenization
* Fallback for Model Optimizer by operation to the legacy Frontend is no longer available.
Fallback by .json config will remain until Model Optimizer is discontinued
(`PR #21523 <https://github.com/openvinotoolkit/openvino/pull/21523>`__).
* Support for the following has been added:
* Mutable variables and resources such as HashTable*, Variable, VariableV2
(`PR #22270 <https://github.com/openvinotoolkit/openvino/pull/22270>`__).
* New tensor types: tf.u16, tf.u32, and tf.u64
(`PR #21864 <https://github.com/openvinotoolkit/openvino/pull/21864>`__).
* 14 NEW Ops*.
`Check the list here (marked as NEW) <https://github.com/openvinotoolkit/openvino/blob/releases/2024/0/src/frontends/tensorflow/docs/supported_ops.md>`__.
* TensorFlow 2.15
(`PR #22180 <https://github.com/openvinotoolkit/openvino/pull/22180>`__).
* The following issues have been fixed:
* UpSampling2D conversion crashed when input type as int16
(`PR #20838 <https://github.com/openvinotoolkit/openvino/pull/20838>`__).
* IndexError list index for Squeeze
(`PR #22326 <https://github.com/openvinotoolkit/openvino/pull/22326>`__).
* Correct FloorDiv computation for signed integers
(`PR #22684 <https://github.com/openvinotoolkit/openvino/pull/22684>`__).
* Fixed bad cast error for tf.TensorShape to ov.PartialShape
(`PR #22813 <https://github.com/openvinotoolkit/openvino/pull/22813>`__).
* Fixed reading tf.string attributes for models in memory
(`PR #22752 <https://github.com/openvinotoolkit/openvino/pull/22752>`__).
*ONNX Framework Support*
* ONNX Frontend now uses the OpenVINO API 2.0.
*PyTorch Framework Support*
* Names for outputs unpacked from dict or tuple are now clearer
(`PR #22821 <https://github.com/openvinotoolkit/openvino/pull/22821>`__).
* FX Graph (torch.compile) now supports kwarg inputs, improving data type coverage.
(`PR #22397 <https://github.com/openvinotoolkit/openvino/pull/22397>`__).
**OpenVINO Model Server**
* OpenVINO™ Runtime backend used is now 2024.0.
* Text generation demo now supports multi batch size, with streaming and unary clients.
* The REST client now supports servables based on mediapipe graphs, including python pipeline
nodes.
* Included dependencies have received security-related updates.
* Reshaping a model in runtime based on the incoming requests (auto shape and auto batch size)
is deprecated and will be removed in the future. Using OpenVINO's dynamic shape models is
recommended instead.
**Neural Network Compression Framework (NNCF)**
* The `Activation-aware Weight Quantization (AWQ) <https://arxiv.org/abs/2306.00978>`__
algorithm for data-aware 4-bit weights compression is now available. It facilitates better
accuracy for compressed LLMs with high ratio of 4-bit weights. To enable it, use the
dedicated ``awq`` optional parameter of ``the nncf.compress_weights()`` API.
* ONNX models are now supported in Post-training Quantization with Accuracy Control, through
the ``nncf.quantize_with_accuracy_control()``, method. It may be used for models in the
OpenVINO IR and ONNX formats.
* A `weight compression example tutorial <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__
is now available, demonstrating how to find the appropriate hyperparameters for the TinyLLama
model from the Hugging Face Transformers, as well as other LLMs, with some modifications.
**OpenVINO Tokenizer**
* Regex support has been improved.
* Model coverage has been improved.
* Tokenizer metadata has been added to rt_info.
* Limited support for Tensorflow Text models has been added: convert MUSE for TF Hub with
string inputs.
* OpenVINO Tokenizers have their own repository now:
`/openvino_tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
**Other Changes and Known Issues**
*Jupyter Notebooks*
The following notebooks have been updated or newly added:
* `Mobile language assistant with MobileVLM <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/279-mobilevlm-language-assistant>`__
* `Depth estimation with DepthAnything <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/280-depth-anything>`__
* `Kosmos-2 <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/281-kosmos2-multimodal-large-language-model>`__
* `Zero-shot Image Classification with SigLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/282-siglip-zero-shot-image-classification>`__
* `Personalized image generation with PhotoMaker <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/283-photo-maker>`__
* `Voice tone cloning with OpenVoice <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/284-openvoice>`__
* `Line-level text detection with Surya <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/285-surya-line-level-text-detection>`__
* `InstantID: Zero-shot Identity-Preserving Generation using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/286-instant-id>`__
* `Tutorial for Big Image Transfer (BIT) model quantization using NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/127-big-transfer-quantization>`__
* `Tutorial for OpenVINO Tokenizers integration into inference pipelines <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/128-openvino-tokenizers>`__
* `LLM chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-llm-chatbot.ipynb>`__ and
`LLM RAG pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-rag-chatbot.ipynb>`__
have received integration with new models: minicpm-2b-dpo, gemma-7b-it, qwen1.5-7b-chat, baichuan2-7b-chat
*Known issues*
| **Component - CPU Plugin**
| *ID* - N/A
| *Description:*
| Starting with 24.0, model inputs and outputs will no longer have tensor names, unless
explicitly set to align with the PyTorch framework behavior.
| **Component - GPU runtime**
| *ID* - 132376
| *Description:*
| First-inference latency slow down for LLMs on Intel® Core™ Ultra processors. Up to 10-20%
drop may occur due to radical memory optimization for processing long sequences
(about 1.5-2 GB reduced memory usage).
| **Component - CPU runtime**
| *ID* - N/A
| *Description:*
| Performance results (first token latency) may vary from those offered by the previous OpenVINO version, for
“latency” hint inference of LLMs with long prompts on Xeon platforms with 2 or more
sockets. The reason is that all CPU cores of just the single socket running the application
are employed, lowering the memory overhead for LLMs when numa control is not used.
| *Workaround:*
| The behavior is expected but stream and thread configuration may be used to include cores
from all sockets.
Deprecation And Support
@ -293,8 +541,8 @@ Discontinued in 2024
* OpenVINO C++/C/Python 1.0 APIs (see
`2023.3 API transition guide <https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html>`__
for reference).
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API)
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API).
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API.
* Tools:
@ -322,6 +570,95 @@ Deprecated and to be removed in the future
:doc:`model conversion transition guide <../documentation/legacy-features/transition-legacy-conversion-api>`.
* OpenVINO property Affinity API will be discontinued with OpenVINO 2025.0.
It will be replaced with CPU binding configurations (``ov::hint::enable_cpu_pinning``).
* OpenVINO Model Server components:
* “auto shape” and “auto batch size” (reshaping a model in runtime) will be removed in the
future. OpenVINO's dynamic shape models are recommended instead.
* The following notebooks have been deprecated and will be removed. For an up-to-date listing
of available notebooks, refer to
`OpenVINO™ Notebook index (openvinotoolkit.github.io) <https://openvinotoolkit.github.io/openvino_notebooks/>`__.
* `Handwritten OCR with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/handwritten-ocr>`__
* See alternative: `Optical Character Recognition (OCR) with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/optical-character-recognition>`__,
* See alternative: `PaddleOCR with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/paddle-ocr-webcam>`__,
* See alternative: `Handwritten Text Recognition Demo <https://docs.openvino.ai/2024/omz_demos_handwritten_text_recognition_demo_python.html>`__
* `Image In-painting with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/image-inpainting>`__
* See alternative: `Image Inpainting Python Demo <https://docs.openvino.ai/2024/omz_demos_image_inpainting_demo_python.html>`__
* `Interactive Machine Translation with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/machine-translation>`__
* See alternative: `Machine Translation Python* Demo <https://docs.openvino.ai/2024/omz_demos_machine_translation_demo_python.html>`__
* `Open Model Zoo Tools Tutorial <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/model-tools>`__
* No alternatives, demonstrates deprecated tools.
* `Super Resolution with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-superresolution>`__
* See alternative: `Super Resolution with PaddleGAN and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-paddlegan-superresolution>`__
* See alternative: `Image Processing C++ Demo <https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/image_processing_demo/cpp/README.md>`__
* `Image Colorization with OpenVINO Tutorial <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-image-colorization>`__
* `Interactive Question Answering with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/question-answering>`__
* See alternative: `BERT Question Answering Embedding Python* Demo <https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/bert_question_answering_embedding_demo/python/README.md>`__
* See alternative: `BERT Question Answering Python* Demo <https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/bert_question_answering_demo/python/README.md>`__
* `Vehicle Detection And Recognition with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vehicle-detection-and-recognition>`__
* See alternative: `Security Barrier Camera C++ Demo <https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/security_barrier_camera_demo/cpp/README.md>`__
* `The attention center model with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/attention-center>`_
* `Image Generation with DeciDiffusion <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/decidiffusion-image-generation>`_
* `Image generation with DeepFloyd IF and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/deepfloyd-if>`_
* `Depth estimation using VI-depth with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/depth-estimation-videpth>`_
* `Instruction following using Databricks Dolly 2.0 and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/dolly-2-instruction-following>`_
* See alternative: `LLM Instruction-following pipeline with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-question-answering>`__
* `Image generation with FastComposer and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/fastcomposer-image-generation>`__
* `Video Subtitle Generation with OpenAI Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/whisper-subtitles-generation>`__
* See alternative: `Automatic speech recognition using Distil-Whisper and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/distil-whisper-asr/distil-whisper-asr.ipynb>`__
* `Introduction to Performance Tricks in OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/performance-tricks>`__
* `Speaker Diarization with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pyannote-speaker-diarization>`__
* `Subject-driven image generation and editing using BLIP Diffusion and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/blip-diffusion-subject-generation>`__
* `Text Prediction with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/text-prediction>`__
* `Training to Deployment with TensorFlow and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/tensorflow-training-openvino>`__
* `Speech to Text with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/speech-to-text>`__
* `Convert and Optimize YOLOv7 with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov7-optimization>`__
* `Quantize Data2Vec Speech Recognition Model using NNCF PTQ API <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/speech-recognition-quantization/speech-recognition-quantization-data2vec.ipynb>`__
* See alternative: `Quantize Speech Recognition Models with accuracy control using NNCF PTQ API <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/quantizing-model-with-accuracy-control/speech-recognition-quantization-wav2vec2.ipynb>`__
* `Semantic segmentation with LRASPP MobileNet v3 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/torchvision-zoo-to-openvino/lraspp-segmentation.ipynb>`__
* `Video Recognition using SlowFast and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/slowfast-video-recognition>`__
* See alternative: `Live Action Recognition with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/action-recognition-webcam>`__
* `Semantic Segmentation with OpenVINO™ using Segmenter <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/segmenter-semantic-segmentation>`__
* `Programming Language Classification with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/code-language-id>`__
* `Stable Diffusion Text-to-Image Demo <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-v2/stable-diffusion-v2-text-to-image-demo.ipynb>`__
* See alternative: `Stable Diffusion v2.1 using Optimum-Intel OpenVINO and multiple Intel Hardware <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-v2/stable-diffusion-v2-optimum-demo.ipynb>`__
* `Text-to-Image Generation with Stable Diffusion v2 and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-v2/stable-diffusion-v2-text-to-image.ipynb>`__
* See alternative: `Stable Diffusion v2.1 using Optimum-Intel OpenVINO and multiple Intel Hardware <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-v2/stable-diffusion-v2-optimum-demo.ipynb>`__
* `Image generation with Segmind Stable Diffusion 1B (SSD-1B) model and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-xl/ssd-b1.ipynb>`__
* `Data Preparation for 2D Medical Imaging <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ct-segmentation-quantize/data-preparation-ct-scan.ipynb>`__
* `Train a Kidney Segmentation Model with MONAI and PyTorch Lightning <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ct-segmentation-quantize/pytorch-monai-training.ipynb>`__
* `Live Inference and Benchmark CT-scan Data with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ct-segmentation-quantize/ct-scan-live-inference.ipynb>`__
* See alternative: `Quantize a Segmentation Model and Show Live Inference <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ct-segmentation-quantize/ct-segmentation-quantize-nncf.ipynb>`__
* `Live Style Transfer with OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/style-transfer-webcam>`__
@ -369,4 +706,3 @@ Performance varies by use, configuration and other factors. Learn more at

186
docs/home.rst Normal file
View File

@ -0,0 +1,186 @@
============================
OpenVINO 2024.1
============================
.. meta::
:google-site-verification: _YqumYQ98cmXUTwtzM_0WIIadtDc6r_TMYGbmGgNvrk
**OpenVINO is an open-source toolkit** for optimizing and deploying deep learning models from cloud
to edge. It accelerates deep learning inference across various use cases, such as generative AI, video,
audio, and language with models from popular frameworks like PyTorch, TensorFlow, ONNX, and more.
Convert and optimize models, and deploy across a mix of Intel® hardware and environments, on-premises
and on-device, in the browser or in the cloud.
Check out the `OpenVINO Cheat Sheet. <https://docs.openvino.ai/2024/_static/download/OpenVINO_Quick_Start_Guide.pdf>`__
.. container::
:name: ov-homepage-banner
.. raw:: html
<link rel="stylesheet" type="text/css" href="_static/css/homepage_style.css">
<div class="line-block">
<section class="splide" aria-label="Splide Banner Carousel">
<div class="splide__track">
<ul class="splide__list">
<li id="ov-homepage-slide1" id class="splide__slide">
<p class="ov-homepage-slide-title">An open-source toolkit for optimizing and deploying deep learning models.</p>
<p class="ov-homepage-slide-subtitle">Boost your AI deep-learning inference performance!</p>
<a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2024/get-started.html">Learn more</a>
</li>
<li id="ov-homepage-slide2" class="splide__slide">
<p class="ov-homepage-slide-title">Better OpenVINO integration with PyTorch!</p>
<p class="ov-homepage-slide-subtitle">Use PyTorch models directly, without converting them first.</p>
<a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2024/openvino-workflow/model-preparation/convert-model-pytorch.html">Learn more</a>
</li>
<li id="ov-homepage-slide3" class="splide__slide">
<p class="ov-homepage-slide-title">OpenVINO via PyTorch 2.0 torch.compile()</p>
<p class="ov-homepage-slide-subtitle">Use OpenVINO directly in PyTorch-native applications!</p>
<a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2024/openvino-workflow/torch-compile.html">Learn more</a>
</li>
<li id="ov-homepage-slide4" class="splide__slide">
<p class="ov-homepage-slide-title">Do you like Generative AI?</p>
<p class="ov-homepage-slide-subtitle">You will love how it performs with OpenVINO!</p>
<a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2024/learn-openvino/interactive-tutorials-python.html">Check out our new notebooks</a>
</li>
<li id="ov-homepage-slide5" id class="splide__slide">
<p class="ov-homepage-slide-title">Boost your AI deep-learning inference performance.</p>
<p class="ov-homepage-slide-subtitle">Use Intel's open-source OpenVINO toolkit for optimizing and deploying deep learning models.</p>
<a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2024/openvino-workflow/running-inference/integrate-openvino-with-your-application.html">Learn more</a>
</li>
</ul>
</div>
</section>
</div>
|
|
.. image:: _static/images/openvino-overview-diagram.jpg
:align: center
:alt: openvino diagram
|
Places to Begin
++++++++++++++++++++++++++++
.. grid:: 2 2 3 3
:class-container: ov-homepage-higlight-grid
.. grid-item-card:: Installation
:img-top: ./_static/images/home_begin_tile_01.png
:class-card: homepage_begin_tile
This guide introduces installation and learning materials for Intel® Distribution of OpenVINO™ toolkit.
.. button-link:: get-started/install-openvino.html
:color: primary
:outline:
Get Started
.. grid-item-card:: Performance Benchmarks
:img-top: ./_static/images/home_begin_tile_02.png
:class-card: homepage_begin_tile
See latest benchmark numbers for OpenVINO and OpenVINO Model Server.
.. button-link:: about-openvino/performance-benchmarks.html
:color: primary
:outline:
View data
.. grid-item-card:: Framework Compatibility
:img-top: ./_static/images/home_begin_tile_03.png
:class-card: homepage_begin_tile
Load models directly (for TensorFlow, ONNX, PaddlePaddle) or convert to OpenVINO format.
.. button-link:: openvino-workflow/model-preparation.html
:color: primary
:outline:
Load your model
.. grid-item-card:: Easy Deployment
:img-top: ./_static/images/home_begin_tile_04.png
:class-card: homepage_begin_tile
Get started in just a few lines of code.
.. button-link:: openvino-workflow/running-inference.html
:color: primary
:outline:
Run Inference
.. grid-item-card:: Serving at scale
:img-top: ./_static/images/home_begin_tile_05.png
:class-card: homepage_begin_tile
Cloud-ready deployments for microservice applications.
.. button-link:: ovms_what_is_openvino_model_server.html
:color: primary
:outline:
Try it out
.. grid-item-card:: Model Compression
:img-top: ./_static/images/home_begin_tile_06.png
:class-card: homepage_begin_tile
Reach for performance with post-training and training-time compression with NNCF.
.. button-link:: openvino-workflow/model-optimization.html
:color: primary
:outline:
Optimize now
|
Key Features
++++++++++++++++++++++++++++
.. grid:: 2 2 2 2
:class-container: homepage_begin_container
.. grid-item-card:: Model Compression
:img-top: ./_static/images/home_key_feature_01.png
:class-card: homepage_begin_key
You can either link directly with OpenVINO Runtime to run inference locally or use OpenVINO Model Server to serve model inference from a separate server or within a Kubernetes environment.
.. grid-item-card:: Fast & Scalable Deployment
:img-top: ./_static/images/home_key_feature_02.png
:class-card: homepage_begin_key
Write an application once, deploy it anywhere, achieving maximum performance from hardware. Automatic device discovery allows for superior deployment flexibility. OpenVINO Runtime supports Linux, Windows, and MacOS and provides Python, C++, and C API. Use your preferred language and OS.
.. grid-item-card:: Lighter Deployment
:img-top: ./_static/images/home_key_feature_03.png
:class-card: homepage_begin_key
Designed with minimal external dependencies reduces the application footprint, simplifying installation and dependency management. Popular package managers enable application dependencies to be easily installed and upgraded. Custom compilation for your specific model(s) further reduces the final binary size.
.. grid-item-card:: Enhanced App Start-Up Time
:img-top: ./_static/images/home_key_feature_04.png
:class-card: homepage_begin_key
In applications where fast start-up is required, OpenVINO significantly reduces first-inference latency by using the CPU for initial inference and then switching to another device once the model has been compiled and loaded to memory. Compiled models are cached, improving start-up time even more.
.. toctree::
:maxdepth: 2
:hidden:
GET STARTED <get-started>
LEARN OPENVINO <learn-openvino>
OPENVINO WORKFLOW <openvino-workflow>
DOCUMENTATION <documentation>
ABOUT OPENVINO <about-openvino>

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-platform_list.pdf
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-Performance-Data.xlsx Normal file
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-platform_list.pdf Normal file
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-system-info-detailed.xlsx Normal file
View File

Binary file not shown.

1019
docs/sphinx_setup/_static/benchmarks_files/OV-benchmark-data.csv
View File

File diff suppressed because it is too large Load Diff