natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] 2024 release notes (#23258)

This commit is contained in:
Karol Blaszczak 2024-03-05 18:28:33 +01:00 committed by GitHub
parent 51769e45ed
commit 4e1d3c8956
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 286 additions and 338 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

598
docs/articles_en/about-openvino/release-notes-openvino.rst
View File

@ -1,370 +1,321 @@
.. {#openvino_release_notes}
OpenVINO Release Notes
========================================
=============================
2023.3 (LTS) - 24.01.2024
###########################
2024.0 - 06 March 2024
#############################
Summary of major features and improvements
++++++++++++++++++++++++++++++++++++++++++++
:doc:`System Requirements <./system-requirements>` | :doc:`Installation Guides <./../get-started/install-openvino>`
What's new
+++++++++++++++++++++++++++++
* More Generative AI coverage and framework integrations to minimize code changes.
* Introducing `OpenVINO Gen AI repository <https://github.com/openvinotoolkit/openvino.genai>`__
on GitHub that demonstrates native C and C++ pipeline samples for Large Language Models
(LLMs). String tensors are now supported as inputs and tokenizers natively to reduce
overhead and ease production.
* New and noteworthy models validated; Mistral, Zephyr, Qwen, ChatGLM3, and Baichuan
* New Jupyter Notebooks for
`Latent Consistency Models (LCM) <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/263-latent-consistency-models-image-generation>`__
and `Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__.
Updated `LLM Chatbot notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot>`__
to include LangChain, Neural Chat, TinyLlama, ChatGLM3, Qwen, Notus and Youri models.
* Torch.compile is now fully integrated with OpenVINO, which now includes a hardware
'options' parameter allowing for seamless inference hardware selection by leveraging
the plugin architecture in OpenVINO.
* Improved out-of-the-box experience for TensorFlow sentence encoding models through the
installation of OpenVINO™ toolkit Tokenizers.
* New and noteworthy models validated:
Mistral, StableLM-tuned-alpha-3b, and StableLM-Epoch-3B.
* OpenVINO™ toolkit now supports Mixture of Experts (MoE), a new architecture that helps
process more efficient generative models through the pipeline.
* JavaScript developers now have seamless access to OpenVINO API. This new binding enables a
smooth integration with JavaScript API.
* Broader Large Language Model (LLM) support and more model compression techniques.
* As part of the Neural Network Compression Framework (NNCF), INT4 weight compression model
formats are now fully supported on Intel® Xeon® CPUs in addition to Intel® Core™ and iGPU,
adding more performance, lower memory usage, and accuracy opportunity when using LLMs.
* Improved performance of transformer based LLM on CPU and GPU using stateful model technique
to increase memory efficiency where internal states are shared among multiple iterations of
inference.
* Easier optimization and conversion of Hugging Face models - compress LLM models to INT8
and INT4 with Hugging Face Optimum command line interface and export models to OpenVINO
format. Note this is part of `Optimum-Intel <https://huggingface.co/docs/optimum/intel/index>`__
which needs to be installed separately.
* Tokenizer and TorchVision transform support is now available in the OpenVINO runtime
(via new API) requiring less preprocessing code and enhancing performance by automatically
handling this model setup. More details on Tokenizers support in Ecosystem section.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Improved quality on INT4 weight compression for LLMs by adding the popular technique,
Activation-aware Weight Quantization, to the Neural Network Compression Framework (NNCF).
This addition reduces memory requirements and helps speed up token generation.
* Experience enhanced LLM performance on Intel® CPUs, with internal memory state enhancement,
and INT8 precision for KV-cache. Specifically tailored for multi-query LLMs like ChatGLM.
* The OpenVINO™ 2024.0 release makes it easier for developers, by integrating more OpenVINO™
features with the Hugging Face ecosystem. Store quantization configurations for popular
models directly in Hugging Face to compress models into INT4 format while preserving
accuracy and performance.
* More portability and performance to run AI at the edge, in the cloud, or locally.
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids).
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Improved performance on ARM platforms using throughput hint, which increases efficiency
in utilization of CPU cores and memory bandwidth.
* Preview JavaScript API to enable node JS development to access JavaScript binding via
source code. See details below.
* Improved `model serving of LLMs <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__
through OpenVINO Model Server. This not only enables LLM serving over KServe v2 gRPC
and REST APIs for more flexibility, but also improves throughput by running processing
like tokenization on the server side. More details in the Ecosystem section.
* A preview plugin architecture of the integrated Neural Processor Unit (NPU) as part of
Intel® Core™ Ultra processor (codename Meteor Lake) is now included in the main OpenVINO™
package on PyPI.
* Improved performance on ARM by enabling the ARM threading library. In addition, we now
support multi-core ARM platforms and enabled FP16 precision by default on MacOS.
* New and improved LLM serving samples from OpenVINO Model Server for multi-batch inputs and
Retrieval Augmented Generation (RAG).
Support Change and Deprecation Notices
++++++++++++++++++++++++++++++++++++++++++
* The OpenVINO™ Development Tools package (pip install openvino-dev) is deprecated and will be
removed from installation options and distribution channels beginning with 2025.0.
For more details, refer to the :doc:`OpenVINO Legacy Features and Components <../documentation/legacy-features>`
page.
* Ubuntu 18.04 support will be discontinued in the 2023.3 LTS release. The recommended version
of Ubuntu is 22.04.
* Starting in release 2023.3 OpenVINO will no longer support Python 3.7 due to the Python
community discontinuing support. Update to a newer version (currently 3.8-3.11) to avoid
interruptions.
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API) will no longer be available in 2024.0 release.
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API will be
discontinued in the 2024.0 release.
* Tools:
* `Deployment Manager <https://docs.openvino.ai/2023.3/openvino_docs_install_guides_deployment_manager_tool.html>`__
is deprecated and will be supported for two years according to our LTS policy.
Visit our :doc:`selector tool <../get-started/install-openvino>` to see
package distribution options or our :doc:`deployment guide <../openvino-workflow/deployment-locally>`
../documentation.
* Accuracy Checker is deprecated and will be discontinued with 2024.0.
* Post-Training Optimization Tool (POT) has been deprecated and the 2023.3 LTS will be
the last release that will support the tool. Developers are encouraged to use the Neural
Network Compression Framework (NNCF) for this feature.
* Model Optimizer is deprecated and will be fully supported until the 2025.0 release.
We encourage developers to perform model conversion through OpenVINO Model Converter
(API call: OVC). Follow the
:doc:`model conversion transition guide <../documentation/legacy-features/transition-legacy-conversion-api>`
for more details.
* Deprecated support for a `git patch <https://github.com/openvinotoolkit/nncf/tree/develop/third_party_integration/huggingface_transformers>`__
for NNCF integration with `huggingface/transformers <https://github.com/huggingface/transformers>`__.
The recommended approach is to use `huggingface/optimum-intel <https://github.com/huggingface/optimum-intel>`__
for applying NNCF optimization on top of models from Hugging Face.
* Support for Apache MXNet, Caffe, and Kaldi model formats is deprecated and will be
discontinued with the 2024.0 release.
* Runtime:
* Intel® Gaussian & Neural Accelerator (Intel® GNA) will be deprecated in a future release.
We encourage developers to use the Neural Processing Unit (NPU) for low-powered systems
like Intel® CoreTM Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs are deprecated and will be discontinued in the 2024.0 release.
Please use API 2.0 in your applications going forward to avoid disruption.
* OpenVINO property Affinity API will be deprecated from 2024.0 and will be discontinued in 2025.0.
It will be replaced with CPU binding configurations (``ov::hint::enable_cpu_pinning``).
OpenVINO™ Development Tools
++++++++++++++++++++++++++++++++++++++++++
* `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__
* Weight compression API, ``nncf.compress_weights()``, has been extended by:
* When using the 'all_layers' parameter, it compresses the model, including embeddings
and final layers, to the 4-bit format. This helps make the model footprint smaller
and improves performance, but it might impact the model accuracy. By default, this
parameter is disabled, and the backup precision (INT8) is assigned for the embeddings
and last layers.
* When using INT8_SYM compression mode for better performance of the compressed model
in case of 8-bit weight compression you might experience an impact on model accuracy
therefore by default, we use INT8_ASYM mode to better balance performance and accuracy.
* We implemented a 4-bit data-aware weight compression feature, introducing the 'dataset'
optional parameter in ``nncf.compress_weights()``. This parameter can be utilized to
mitigate accuracy loss in compressed models. It's important to note that enabling
this option will extend the compression time.
* Post-training Quantization with Accuracy Control, ``nncf.quantize_with_accuracy_control()``,
has been extended by the 'restore_mode' optional parameter to revert weights to INT8
instead of the original precision. This parameter helps to reduce the size of the
quantized model and improves its performance. By default, it is disabled and model
weights are reverted to the original precision in ``nncf.quantize_with_accuracy_control()``.
OpenVINO™ Runtime
++++++++++++++++++++++++
+++++++++++++++++++++++++++++
* Model Import Updates
Common
-----------------------------
* TensorFlow Framework Support
* Supported TF1 While Control flow construction w/o TensorArray operations
(`PR #20800 <https://github.com/openvinotoolkit/openvino/pull/20800>`__).
* Support for complex tensors has been added
(`PR #20860 <https://github.com/openvinotoolkit/openvino/pull/20860>`__),
(`PR #21477 <https://github.com/openvinotoolkit/openvino/pull/21477>`__).
* Provided fixes for the following:
* Accept any model file extension for frozen protobuf format
(`PR #21508 <https://github.com/openvinotoolkit/openvino/pull/21508>`__).
* Correct ArgMin/ArgMax translators for repeating elements case
(`PR #21364 <https://github.com/openvinotoolkit/openvino/pull/21364>`__).
* Correct PartitionedCall translator when numbers of external and internal
body inputs mismatch
(`PR #20825 <https://github.com/openvinotoolkit/openvino/pull/20825>`__).
* PyTorch Framework Support
* Added support of nested dictionaries and lists as example input.
* Disabled ``torch.jit.freeze`` in default model tracing scenario and
improved support for models without freezing, extending model
coverage and improving accuracy for some models.
* ONNX Framework Support
* Switched to ONNX 1.15.0 as a supported version of original framework
(`PR #20929 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* CPU
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids)
with sub-numa (SNC) and efficient core resource scheduling to improve performance.
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Further improved performance of LLMs in INT4 weight compression, especially on 1st
token latency and on 4th and 5th Gen of Intel Xeon platforms (codename Sapphire
Rapids and Emerald Rapids) with AMX capabilities.
* Improved performance of transformer-based LLM using stateful model technique to
increase memory efficiency where internal states (KV cache) are shared among multiple
iterations of inference. The stateful model implementation supports both greedy search
and beam search (preview) for LLMs. This technique also reduces the memory footprint
of LLMs, where Intel Core and Ultra platforms like Raptor Lake and Meteor Lake can
run INT4 models, such as Llama v2 7B.
* Improved performance on ARM platforms with throughput hint, by increasing
efficiency in usage of the CPU cores and memory bandwidth.
* GPU
* Full support for Intel® Core™ Ultra (codename Meteor Lake) integrated graphics.
* For LLMs, the first inference latency for INT8 and INT4 weight-compressed models has
been improved on iGPU thanks to more efficient context processing. Overall average
token latency for INT8 and INT4 has also been enhanced on iGPU with graph compilation
optimization, various host overhead optimization, and dynamic padding support for GEMM.
* Stateful model is functionally supported for LLMs.
* Model caching for dynamically shaped models is now supported. Model loading time is
improved for these models, including LLMs.
* API for switching between size mode (model caching) and speed mode (kernel caching)
is introduced.
* The model cache file name is changed to be independent of GPU driver versions.
The GPU will not generate separate model cache files when the driver is updated.
* Compilation time for Stable Diffusion models has been improved.
* NPU
* NPU plugin is available as part of OpenVINO. With the Intel(R) Core Ultra NPU driver
installed, inference can run on the NPU device.
* AUTO device plug-in (AUTO)
* Introduced the round-robin policy to AUTO cumulative throughput hint, which dispatches
inference requests to multiple devices (such as multiple GPU devices) in the round-robin
sequence, instead of in the device priority sequence. The device priority sequence
remains as the default configuration.
* AUTO loads stateful models to GPU or CPU per device priority, since GPU now supports
stateful model inference.
* OpenVINO Common
* Enhanced support of String tensors has been implemented, enabling the use of operators
and models that rely on string tensors. This update also enhances the capability in
the torchvision preprocessing (`PR #21244 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* A new feature has been added that enables the selection of P-Cores for model compilation
on CPU device(s) with hybrid architecture (i.e. Intel® Core™ 12th Gen and beyond).
This will reduce compilation time compared to previous implementation where P-cores
and E-cores are used randomly by OS scheduling.
* OpenVINO JavaScript API (preview feature)
* We've introduced a preview version of
`JS API <https://github.com/openvinotoolkit/openvino/tree/master/src/bindings/js>`__
for OpenVINO runtime in this release. We hope that you will try this feature and
provide your feedback through GitHub `issues <https://github.com/openvinotoolkit/openvino/issues>`__.
* Known limitations:
* Only supported in manylinux and x86 (Windows, ARM, ARM64, and macOS have not been tested)
* Node.js version >= 18.16
* CMake version < 3.14 is not supported
* gcc compiler version < 7 is not supported
* OpenVINO Python API
* Introducing string tensor support for Python API.
* Added support for the following:
* Create ov.Tensor from Python lists
* Create ov.Tensor from empty numpy arrays.
* Constants from empty numpy arrays.
* Autogenerated get/set methods for Node attributes.
* Inference functions (``InferRequest.infer/start_async``, ``CompiledModel.__call__`` etc.) support OVDict as the input.
* PILLOW interpolation modes bindings. (`PR #21188 <https://github.com/openvinotoolkit/openvino/pull/21188>`__ external contribution: @meetpatel0963)
* Torchvision to :doc:`OpenVINO preprocessing <../openvino-workflow/running-inference/string-tensors>`
converter ../documentation has been added to OpenVINO docs.
* The legacy API for CPP and Python bindings has been removed.
* StringTensor support has been extended by operators such as ``Gather``, ``Reshape``, and
``Concat``, as a foundation to improve support for tokenizer operators and compliance with
the TensorFlow Hub.
* oneDNN has been updated to v3.3.
(`see oneDNN release notes <https://github.com/oneapi-src/oneDNN/releases>`__).
OpenVINO Ecosystem
+++++++++++++++++++++++++++++++++++++++++++++
CPU Device Plugin
-----------------------------
* OpenVINO Tokenizer (Preview feature)
* LLM performance on Intel® CPU platforms has been improved for systems based on AVX2 and
AVX512, using dynamic quantization and internal memory state optimization, such as INT8
precision for KV-cache. 13th and 14th generations of Intel® Core™ processors and Intel® Core™
Ultra processors use AVX2 for CPU execution, and these platforms will benefit from speedup.
Enable these features by setting ``"DYNAMIC_QUANTIZATION_GROUP_SIZE":"32"`` and
``"KV_CACHE_PRECISION":"u8"`` in the configuration file.
* The ``ov::affinity`` API configuration is now deprecated and will be removed in release
2025.0.
* The following have been improved and optimized:
* OpenVINO Tokenizer adds text processing operations to OpenVINO:
* Multi-query structure LLMs (such as ChatGLM 2/3) for BF16 on the 4th and 5th generation
Intel® Xeon® Scalable processors.
* `Mixtral <https://huggingface.co/docs/transformers/model_doc/mixtral>`__ model performance.
* 8-bit compressed LLM compilation time and memory usage, valuable for models with large
embeddings like `Qwen <https://github.com/QwenLM/Qwen>`__.
* Convolutional networks in FP16 precision on ARM platforms.
* Text PrePostprocessing without third-party dependencies
* Convert a HuggingFace tokenizer into the OpenVINO model tokenizer and the
detokenizer using a CLI tool or Python API
* Connect a tokenizer and a model to get a single model with text input
GPU Device Plugin
-----------------------------
* OpenVINO Tokenizer models work only on the CPU device
* Supported platforms: Linux (x86 and ARM), Windows and Mac (x86 and ARM)
* The following have been improved and optimized:
* Average token latency for LLMs on integrated GPU (iGPU) platforms, using INT4-compressed
models with large context size on Intel® Core™ Ultra processors.
* LLM beam search performance on iGPU. Both average and first-token latency decrease may be
expected for larger context sizes.
* Multi-batch performance of YOLOv5 on iGPU platforms.
* Memory usage for LLMs has been optimized, enabling '7B' models with larger context on
16Gb platforms.
NPU Device Plugin (preview feature)
-----------------------------------
* The NPU plugin for OpenVINO™ is now available through PyPI (run “pip install openvino”).
OpenVINO Python API
-----------------------------
* ``.add_extension`` method signatures have been aligned, improving API behavior for better
user experience.
OpenVINO C API
-----------------------------
* ov_property_key_cache_mode (C++ ov::cache_mode) now enables the ``optimize_size`` and
``optimize_speed`` modes to set/get model cache.
* The VA surface on Windows exception has been fixed.
OpenVINO Node.js API
-----------------------------
* OpenVINO - `JS bindings <https://docs.openvino.ai/2024/api/nodejs_api/nodejs_api.html>`__
are consistent with the OpenVINO C++ API.
* A new distribution channel is now available: Node Package Manager (npm) software registry
(:doc:`check the installation guide <../get-started/install-openvino/install-openvino-npm>`).
* JavaScript API is now available for Windows users, as some limitations for platforms other
than Linux have been removed.
TensorFlow Framework Support
-----------------------------
* String tensors are now natively supported, handled on input, output, and intermediate layers
(`PR #22024 <https://github.com/openvinotoolkit/openvino/pull/22024>`__).
* TensorFlow Hub universal-sentence-encoder-multilingual inferred out of the box
* string tensors supported for ``Gather``, ``Concat``, and ``Reshape`` operations
* integration with openvino-tokenizers module - importing openvino-tokenizers automatically
patches TensorFlow FE with the required translators for models with tokenization
* Fallback for Model Optimizer by operation to the legacy Frontend is no longer available.
Fallback by .json config will remain until Model Optimizer is discontinued
(`PR #21523 <https://github.com/openvinotoolkit/openvino/pull/21523>`__).
* Support for the following has been added:
* Mutable variables and resources such as HashTable*, Variable, VariableV2
(`PR #22270 <https://github.com/openvinotoolkit/openvino/pull/22270>`__).
* New tensor types: tf.u16, tf.u32, and tf.u64
(`PR #21864 <https://github.com/openvinotoolkit/openvino/pull/21864>`__).
* 14 NEW Ops*.
`Check the list here (marked as NEW) <https://github.com/openvinotoolkit/openvino/blob/releases/2024/0/src/frontends/tensorflow/docs/supported_ops.md>`__.
* TensorFlow 2.15
(`PR #22180 <https://github.com/openvinotoolkit/openvino/pull/22180>`__).
* The following issues have been fixed:
* UpSampling2D conversion crashed when input type as int16
(`PR #20838 <https://github.com/openvinotoolkit/openvino/pull/20838>`__).
* IndexError list index for Squeeze
(`PR #22326 <https://github.com/openvinotoolkit/openvino/pull/22326>`__).
* Correct FloorDiv computation for signed integers
(`PR #22684 <https://github.com/openvinotoolkit/openvino/pull/22684>`__).
* Fixed bad cast error for tf.TensorShape to ov.PartialShape
(`PR #22813 <https://github.com/openvinotoolkit/openvino/pull/22813>`__).
* Fixed reading tf.string attributes for models in memory
(`PR #22752 <https://github.com/openvinotoolkit/openvino/pull/22752>`__).
* OpenVINO Model Server
ONNX Framework Support
-----------------------------
* Added support for serving pipelines with custom nodes implemented as a
`python code <https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/quickstart.md>`__
This greatly simplifies exposing GenAI algorithms based on Hugging Face
and Optimum libraries. It can be also applied for arbitrary pre and
post-processing in model serving pipelines.
* Included a new set of model serving demos that use custom nodes with python
code. These include LLM `text generation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__,
`stable diffusion <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/stable_diffusion>`__,
and `seq2seq translation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/seq2seq_translation>`__.
* Improved video stream analysis `demo <https://github.com/openvinotoolkit/model_server/tree/main/demos/real_time_stream_analysis/python>`__.
A simple client example can now process the
video stream from a local camera, video file or RTSP stream.
* Learn more about these changes on
`GitHub <https://github.com/openvinotoolkit/model_server/releases>`__.
* ONNX Frontend now uses the OpenVINO API 2.0.
PyTorch Framework Support
-----------------------------
* Names for outputs unpacked from dict or tuple are now clearer
(`PR #22821 <https://github.com/openvinotoolkit/openvino/pull/22821>`__).
* FX Graph (torch.compile) now supports kwarg inputs, improving data type coverage.
(`PR #22397 <https://github.com/openvinotoolkit/openvino/pull/22397>`__).
* Jupyter Notebook Tutorials
OpenVINO Model Server
+++++++++++++++++++++++++++++
* The following notebooks have been updated or newly added:
* `Sound generation with AudioLDM2 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/270-sound-generation-audioldm2>`__.
* `Single-step image generation using SDXL-turbo and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/271-sdxl-turbo>`__.
* `Paint by Example using Diffusion models and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/272-paint-by-example>`__.
* `LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/273-stable-zephyr-3b-chatbot>`__.
* `Object segmentations with EfficientSAM and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/274-efficient-sam>`__.
* `Create an LLM-powered RAG system using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-rag-chatbot.ipynb>`__
- Demonstrates an integration with LangChain.
* `High-resolution image generation with Segmind-VegaRT and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/248-stable-diffusion-xl/248-segmind-vegart.ipynb>`__.
* `Text-to-Image Generation with LCM LoRA and ControlNet Conditioning <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/263-latent-consistency-models-image-generation/263-lcm-lora-controlnet.ipynb>`__.
* `LLM Instruction-following pipeline with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/275-llm-question-answering>`__ -
Demonstrates how to run an instruction-following text generation pipeline using
tiny-llama-1b-chat, phi-2, dolly-v2-3b, red-pajama-3b-instruct and mistral-7b models.
* `LLM chabot notebook <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot>`__
updated with support for new LLMs and INT4/INT8 Weight Compression: TinyLlama-1b-chat,
Mistral-7B, neural-chat-7b, notus-7b, ChatGLM3, youri-7b-chat (for Japanese language).
* Added optimization support (8-bit quantization, weight compression) by NNCF for the following notebooks:
* `Image generation with Würstchen and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/265-wuerstchen-image-generation>`__
* `QR-code monster <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/264-qrcode-monster>`__
* `INT4-compression support for LLaVA multimodal chatbot <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/257-llava-multimodal-chatbot>`__
* `Distil-whisper quantization <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__
* OpenVINO™ Runtime backend used is now 2024.0.
* Text generation demo now supports multi batch size, with streaming and unary clients.
* The REST client now supports servables based on mediapipe graphs, including python pipeline
nodes.
* Included dependencies have received security-related updates.
* Reshaping a model in runtime based on the incoming requests (auto shape and auto batch size)
is deprecated and will be removed in the future. Using OpenVINO's dynamic shape models is
recommended instead.
Neural Network Compression Framework (NNCF)
+++++++++++++++++++++++++++++++++++++++++++
* The `Activation-aware Weight Quantization (AWQ) <https://arxiv.org/abs/2306.00978>`__
algorithm for data-aware 4-bit weights compression is now available. It facilitates better
accuracy for compressed LLMs with high ratio of 4-bit weights. To enable it, use the
dedicated ``awq`` optional parameter of ``the nncf.compress_weights()`` API.
* ONNX models are now supported in Post-training Quantization with Accuracy Control, through
the ``nncf.quantize_with_accuracy_control()``, method. It may be used for models in the
OpenVINO IR and ONNX formats.
* A `weight compression example tutorial <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__
is now available, demonstrating how to find the appropriate hyperparameters for the TinyLLama
model from the Hugging Face Transformers, as well as other LLMs, with some modifications.
OpenVINO Tokenizer
+++++++++++++++++++++++++++++
* Regex support has been improved.
* Model coverage has been improved.
* Tokenizer metadata has been added to rt_info.
* Limited support for Tensorflow Text models has been added: convert MUSE for TF Hub with
string inputs.
* OpenVINO Tokenizers have their own repository now:
`/openvino_tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
Other Changes and Known Issues
+++++++++++++++++++++++++++++++
Jupyter Notebooks
-----------------------------
The following notebooks have been updated or newly added:
* `Mobile language assistant with MobileVLM <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/279-mobilevlm-language-assistant>`__
* `Depth estimation with DepthAnything <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/280-depth-anything>`__
* `Kosmos-2 <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/281-kosmos2-multimodal-large-language-model>`__
* `Zero-shot Image Classification with SigLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/282-siglip-zero-shot-image-classification>`__
* `Personalized image generation with PhotMaker <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/283-photo-maker>`__
* `Voice tone cloning with OpenVoice <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/284-openvoice>`__
* `Line-level text detection with Surya <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/285-surya-line-level-text-detection>`__
* `InstantID: Zero-shot Identity-Preserving Generation using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/286-instant-id>`__
* `Tutorial for Big Image Transfer (BIT) model quantization using NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/127-big-transfer-quantization>`__
* `Tutorial for OpenVINO Tokenizers integration into inference pipelines <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/128-openvino-tokenizers>`__
* `LLM chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-llm-chatbot.ipynb>`__ and
`LLM RAG pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-rag-chatbot.ipynb>`__
have received integration with new models: minicpm-2b-dpo, gemma-7b-it, qwen1.5-7b-chat, baichuan2-7b-chat
Known issues
++++++++++++++++++++++++++++++++++++++++++++
-----------------------------
| **ID - 127202**
| *Component* - CPU Plugin
| **Component - CPU Plugin**
| *ID* - N/A
| *Description:*
| Deeplabv3 model from TF framework shows lower performance than previous
release. This is because the TopK layer in the model is now correctly
conducting the stable sort as specified by the model, slower than the
previous unstable sort.
| *Workaround:*
| This release has the correct behavior. If performance is critical,
please use the previous version of OpenVINO, or tune the model.
| Starting with 24.0, model inputs and outputs will no longer have tensor names, unless
explicitly set to align with the PyTorch framework behavior.
| **ID - 123101**
| *Component* - GPU plugin
| **Component - GPU runtime**
| *ID* - 132376
| *Description:*
| Hung up of GPU plugin on A770 Graphics (dGPU) in case of large
batch size (1750).
| First-inference latency slow down for LLMs on Intel® Core™ Ultra processors. Up to 10-20%
drop may occur due to radical memory optimization for processing ling sequences
(about 1.5-2 GB reduced memory usage).
| **Component - CPU runtime**
| *ID* - N/A
| *Description:*
| Performance results (first token latency) may vary from those offered by the previous OpenVINO version, for
“latency” hint inference of LLMs with long prompts on Xeon platforms with 2 or more
sockets. The reason is that all CPU cores of just the single socket running the application
are employed, lowering the memory overhead for LLMs when numa control is not used.
| *Workaround:*
| Decrease the batch size, and wait for the fixed driver released.
| The behavior is expected but stream and thread configuration may be used to include cores
from all sockets.
Included in This Release
+++++++++++++++++++++++++++++++++++++++++++++
Deprecation And Support
+++++++++++++++++++++++++++++
Using deprecated features and components is not advised. They are available to enable a smooth
transition to new solutions and will be discontinued in the future. To keep using discontinued
features, you will have to revert to the last LTS OpenVINO version supporting them.
For more details, refer to the :doc:`OpenVINO Legacy Features and Components <../documentation/legacy-features>`
page.
The Intel® Distribution of OpenVINO™ toolkit is available for downloading in
three types of operating systems: Windows, Linux, and macOS.
Discontinued in 2024
-----------------------------
* Runtime components:
* Intel® Gaussian & Neural Accelerator (Intel® GNA). Consider using the Neural Processing
Unit (NPU) for low-powered systems like Intel® Core™ Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs (see
`2023.3 API transition guide <https://docs.openvino.ai/2023.3/openvino_2_0_transition_guide.html>`__
for reference).
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API)
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API
* Tools:
* Deployment Manager. See :doc:`installation <../get-started/install-openvino>` and
:doc:`deployment <../get-started/install-openvino>` guides for current distribution
options.
* `Accuracy Checker <https://docs.openvino.ai/2023.3/omz_tools_accuracy_checker.html>`__.
* `Post-Training Optimization Tool <https://docs.openvino.ai/2023.3/pot_introduction.html>`__
(POT). Neural Network Compression Framework (NNCF) should be used instead.
* A `Git patch <https://github.com/openvinotoolkit/nncf/tree/develop/third_party_integration/huggingface_transformers>`__
for NNCF integration with `huggingface/transformers <https://github.com/huggingface/transformers>`__.
The recommended approach is to use `huggingface/optimum-intel <https://github.com/huggingface/optimum-intel>`__
for applying NNCF optimization on top of models from Hugging Face.
* Support for Apache MXNet, Caffe, and Kaldi model formats. Conversion to ONNX may be used
as a solution.
Deprecated and to be removed in the future
--------------------------------------------
* The OpenVINO™ Development Tools package (pip install openvino-dev) will be removed from
installation options and distribution channels beginning with OpenVINO 2025.
* Model Optimizer will be discontinued with OpenVINO 2025.0. Consider using the
:doc:`new conversion methods <../openvino-workflow/model-preparation/convert-model-to-ir>`
instead. For more details, see the
:doc:`model conversion transition guide <../documentation/legacy-features/transition-legacy-conversion-api>`.
* OpenVINO property Affinity API will be discontinued with OpenVINO 2025.0.
It will be replaced with CPU binding configurations (``ov::hint::enable_cpu_pinning``).
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| Component || License | Location |
+================================+===================================+=================+=================+=======================+=================================================+
|| OpenVINO (Inference Engine) C++ Runtime || Dual licensing: || <install_root>/runtime/* |
|| Unified API to integrate the inference with application logic || Intel® OpenVINO™ Distribution License (Version May 2021) || <install_root>/runtime/include/* |
|| OpenVINO (Inference Engine) Headers || Apache 2.0 || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Python API || Apache 2.0 || <install_root>/python/* |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Samples || Apache 2.0 || <install_root>/samples/* |
|| Samples that illustrate OpenVINO C++/ Python API usage || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| [Deprecated] Deployment manager || Apache 2.0 || <install_root>/tools/deployment_manager/* |
|| The Deployment Manager is a Python command-line tool that || || |
|| creates a deployment package by assembling the model, IR files, || || |
|| your application, and associated dependencies into a runtime || || |
|| package for your target device. || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
Legal Information
@ -407,12 +358,7 @@ For more complete information about compiler optimizations, see our Optimization
Performance varies by use, configuration and other factors. Learn more at
`www.Intel.com/PerformanceIndex <www.Intel.com/PerformanceIndex>`__.
Download
+++++++++++++++++++++++++++++++++++++++++++++
`The OpenVINO product selector tool <https://docs.openvino.ai/install>`__
provides easy access to the right packages that match your desired OS, version,
and distribution options.

26
docs/articles_en/about-openvino/system-requirements.rst
View File

@ -39,7 +39,6 @@ CPU
* CentOS 7
* Red Hat Enterprise Linux 8, 64-bit
* Ubuntu 18 ARM64
* Debian 9 ARM
GPU
##########
@ -71,14 +70,15 @@ GPU
* The use of GPU requires drivers that are not included in the Intel®
Distribution of OpenVINO™ toolkit package.
* Processor graphics are not included in all processors. See
`Product Specifications <https://ark.intel.com/>`__
`Product Specifications <https://ark.intel.com/>`__
for information about your processor.
* While this release of OpenVINO supports Ubuntu 20.04, the driver stack
for Intel discrete graphic cards does not fully support Ubuntu 20.04.
We recommend using Ubuntu 22.04 when executing on discrete graphics.
* The following minimum (i.e., used for old hardware) OpenCL™ driver's versions
were used during OpenVINO internal validation: 22.43 for Ubuntu 22.04, 21.48
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8.
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8 (some hardware may require
higher versions of drivers).
Intel® Neural Processing Unit
################################
@ -130,7 +130,7 @@ Operating systems and developer environment
* GNU Compiler Collection and CMake are needed for building from source:
* `GNU Compiler Collection (GCC) <https://www.gnu.org/software/gcc/>`__ 7.5 and above
* `CMake <https://cmake.org/download/>`__ 3.10 or higher
* `CMake <https://cmake.org/download/>`__ 3.13 or higher
Higher versions of kernel might be required for 10th Gen Intel® Core™ Processors, 11th Gen
Intel® Core™ Processors, 11th Gen Intel® Core™ Processors S-Series Processors, 12th Gen
@ -143,8 +143,8 @@ Operating systems and developer environment
Build environment components:
* `Microsoft Visual Studio 2019 <https://visualstudio.microsoft.com/vs/older-downloads/>`__
* `CMake <https://cmake.org/download/>`__ 3.10 or higher
* `Python* 3.8-3.11 <http://www.python.org/downloads/>`__
* `CMake <https://cmake.org/download/>`__ 3.16 or higher
* `Python <http://www.python.org/downloads/>`__ 3.8-3.11
* `Intel® HD Graphics Driver <https://downloadcenter.intel.com/product/80939/Graphics-Drivers>`__
required for inference on GPU
@ -154,15 +154,15 @@ Operating systems and developer environment
Build environment components:
* `Xcode* 10.3 <https://developer.apple.com/xcode/>`__
* `Python* 3.8-3.11 <http://www.python.org/downloads/>`__
* `CMake <https://cmake.org/download/>`__ 3.10 or higher
* `Xcode <https://developer.apple.com/xcode/>`__ 10.3
* `CMake <https://cmake.org/download/>`__ 3.13 or higher
* `Python <http://www.python.org/downloads/>`__ 3.8-3.11
.. tab-item:: DL frameworks versions:
* TensorFlow 1.15, 2.12
* ONNX 1.14.1
* PaddlePaddle 2.4
* TensorFlow 1.15, 2.13.1
* ONNX 1.15
* PaddlePaddle 2.5
This package can be installed on other versions of DL Frameworks
but only the versions specified here are fully validated.
@ -178,6 +178,8 @@ Operating systems and developer environment
such as 12th Gen Intel Core and above.
Legal Information
+++++++++++++++++++++++++++++++++++++++++++++