natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] releasenotes 23.3 update (#22362)

This commit is contained in:
Karol Blaszczak 2024-01-24 08:34:59 +01:00 committed by GitHub
parent 7721fa7603
commit 0245affbb4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 360 additions and 317 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

553
docs/articles_en/about_openvino/releasenotes_for_openvino.rst
View File

@ -1,314 +1,347 @@
.. {#openvino_release_notes}
OpenVINO Release Notes
======================
========================================
The Intel® Distribution of OpenVINO™ toolkit is an open-source solution for optimizing
and deploying AI inference in domains such as computer vision,automatic speech
recognition, natural language processing, recommendation systems, and generative AI.
With its plug-in architecture, OpenVINO enables developers to write once and deploy
anywhere. We are proud to announce the release of OpenVINO 2023.2 introducing a range
of new features, improvements, and deprecations aimed at enhancing the developer
experience.
New and changed in 2023.2
2023.3 (LTS) - 24.01.2024
###########################
Summary of major features and improvements
++++++++++++++++++++++++++++++++++++++++++++
* More Generative AI coverage and framework integrations to minimize code changes.
* More Generative AI coverage and framework integrations to minimize code changes.
* **Expanded model support for direct PyTorch model conversion** - automatically convert
additional models directly from PyTorch or execute via ``torch.compile`` with OpenVINO
as the backend.
* **New and noteworthy models supported** - we have enabled models used for chatbots,
instruction following, code generation, and many more, including prominent models
like Llava, chatGLM, Bark (text to audio) and LCM (Latent Consistency Models, an
optimized version of Stable Diffusion).
* **Easier optimization and conversion of Hugging Face models** - compress LLM models
to Int8 with the Hugging Face Optimum command line interface and export models to
the OpenVINO IR format.
* **OpenVINO is now available on Conan** - a package manager which allows more seamless
package management for large scale projects for C and C++ developers.
* Introducing `OpenVINO Gen AI repository <https://github.com/openvinotoolkit/openvino.genai>`__
on GitHub that demonstrates native C and C++ pipeline samples for Large Language Models
(LLMs). String tensors are now supported as inputs and tokenizers natively to reduce
overhead and ease production.
* New and noteworthy models validated; Mistral, Zephyr, Qwen, ChatGLM3, and Baichuan
* New Jupyter Notebooks for
`Latent Consistency Models (LCM) <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/263-latent-consistency-models-image-generation>`__
and `Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__.
Updated `LLM Chatbot notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot>`__
to include LangChain, Neural Chat, TinyLlama, ChatGLM3, Qwen, Notus and Youri models.
* Torch.compile is now fully integrated with OpenVINO, which now includes a hardware
'options' parameter allowing for seamless inference hardware selection by leveraging
the plugin architecture in OpenVINO.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Accelerate inference for LLM models on Intel® CoreTM CPU and iGPU with the
use of Int8 model weight compression.
* Expanded model support for dynamic shapes for improved performance on GPU.
* Preview support for Int4 model format is now included. Int4 optimized model
weights are now available to try on Intel® Core™ CPU and iGPU, to accelerate
models like Llama 2 and chatGLM2.
* The following Int4 model compression formats are supported for inference
in runtime:
* Generative Pre-training Transformer Quantization (GPTQ); with GPTQ-compressed
models, you can access them through the Hugging Face repositories.
* Native Int4 compression through Neural Network Compression Framework (NNCF).
* As part of the Neural Network Compression Framework (NNCF), INT4 weight compression model
formats are now fully supported on Intel® Xeon® CPUs in addition to Intel® Core™ and iGPU,
adding more performance, lower memory usage, and accuracy opportunity when using LLMs.
* Improved performance of transformer based LLM on CPU and GPU using stateful model technique
to increase memory efficiency where internal states are shared among multiple iterations of
inference.
* Easier optimization and conversion of Hugging Face models - compress LLM models to INT8
and INT4 with Hugging Face Optimum command line interface and export models to OpenVINO
format. Note this is part of `Optimum-Intel <https://huggingface.co/docs/optimum/intel/index>`__
which needs to be installed separately.
* Tokenizer and TorchVision transform support is now available in the OpenVINO runtime
(via new API) requiring less preprocessing code and enhancing performance by automatically
handling this model setup. More details on Tokenizers support in Ecosystem section.
* More portability and performance to run AI at the edge, in the cloud, or locally.
* **In 2023.1 we announced full support for ARM** architecture, now we have improved
performance by enabling FP16 model formats for LLMs and integrating additional
acceleration libraries to improve latency.
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids).
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Improved performance on ARM platforms using throughput hint, which increases efficiency
in utilization of CPU cores and memory bandwidth.
* Preview JavaScript API to enable node JS development to access JavaScript binding via
source code. See details below.
* Improved `model serving of LLMs <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__
through OpenVINO Model Server. This not only enables LLM serving over KServe v2 gRPC
and REST APIs for more flexibility, but also improves throughput by running processing
like tokenization on the server side. More details in the Ecosystem section.
Support Change and Deprecation Notices
++++++++++++++++++++++++++++++++++++++++++
* The OpenVINO™ Development Tools package (pip install openvino-dev) is deprecated
and will be removed from installation options and distribution channels with
2025.0. To learn more, refer to the
:doc:`OpenVINO Legacy Features and Components page <openvino_legacy_features>`.
To ensure optimal performance, install the OpenVINO package (pip install openvino),
which includes essential components such as OpenVINO Runtime, OpenVINO Converter,
and Benchmark Tool.
* The OpenVINO™ Development Tools package (pip install openvino-dev) is deprecated and will be
removed from installation options and distribution channels beginning with 2025.0.
For more details, refer to the :doc:`OpenVINO Legacy Features and Components <openvino_legacy_features>`
page.
* Ubuntu 18.04 support will be discontinued in the 2023.3 LTS release. The recommended version
of Ubuntu is 22.04.
* Starting in release 2023.3 OpenVINO will no longer support Python 3.7 due to the Python
community discontinuing support. Update to a newer version (currently 3.8-3.11) to avoid
interruptions.
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API) will no longer be available in
2024.0 release.
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API will be
discontinued in the 2024.0 release.
* Tools:
* :doc:`Deployment Manager <openvino_docs_install_guides_deployment_manager_tool>`
is deprecated and will be removed in the 2024.0 release.
is deprecated and will be supported for two years according to our LTS policy.
Visit our :doc:`selector tool <openvino_docs_install_guides_overview>` to see
package distribution options or our :doc:`deployment guide <openvino_deployment_guide>`
documentation.
* Accuracy Checker is deprecated and will be discontinued with 2024.0.
* Post-Training Optimization Tool (POT) is deprecated and will be
discontinued with 2024.0.
* Model Optimizer is deprecated and will be fully supported up until the 2025.0
release. Model conversion to the OpenVINO format should be performed through
OpenVINO Model Converter, which is part of the PyPI package. Follow the
:doc:`Model Optimizer to OpenVINO Model Converter transition <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`
guide for smoother transition. Known limitations are TensorFlow model with
TF1 Control flow and object detection models. These limitations relate to
the gap in TensorFlow direct conversion capabilities which will be addressed
in upcoming releases.
* PyTorch 1.13 support is deprecated in Neural Network Compression Framework (NNCF)
* Post-Training Optimization Tool (POT) has been deprecated and the 2023.3 LTS will be
the last release that will support the tool. Developers are encouraged to use the Neural
Network Compression Framework (NNCF) for this feature.
* Model Optimizer is deprecated and will be fully supported until the 2025.0 release.
We encourage developers to perform model conversion through OpenVINO Model Converter
(API call: OVC). Follow the
:doc:`model conversion transition guide <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`
for more details.
* Deprecated support for a `git patch <https://github.com/openvinotoolkit/nncf/tree/develop/third_party_integration/huggingface_transformers>`__
for NNCF integration with `huggingface/transformers <https://github.com/huggingface/transformers>`__.
The recommended approach is to use `huggingface/optimum-intel <https://github.com/huggingface/optimum-intel>`__
for applying NNCF optimization on top of models from Hugging Face.
* Support for Apache MXNet, Caffe, and Kaldi model formats is deprecated and will be
discontinued with the 2024.0 release.
* Runtime:
* Intel® Gaussian & Neural Accelerator (Intel® GNA) will be deprecated in a future
release. We encourage developers to use the Neural Processing Unit (NPU) for
low powered systems like Intel® Core™ Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs will be discontinued with 2024.0.
* Python 3.7 support has been discontinued.
* Intel® Gaussian & Neural Accelerator (Intel® GNA) will be deprecated in a future release.
We encourage developers to use the Neural Processing Unit (NPU) for low-powered systems
like Intel® CoreTM Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs are deprecated and will be discontinued in the 2024.0 release.
Please use API 2.0 in your applications going forward to avoid disruption.
* OpenVINO property Affinity API will be deprecated from 2024.0 and will be discontinued in 2025.0.
It will be replaced with CPU binding configurations (``ov::hint::enable_cpu_pinning``).
OpenVINO™ Development Tools
++++++++++++++++++++++++++++++++++++++++++
List of components and their changes:
------------------------------------------
* :doc:`OpenVINO Model Converter tool <openvino_docs_model_processing_introduction>`
now supports the original framework shape format.
* `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__
* Added data-free Int4 weight compression support for LLMs in OpenVINO IR with
``nncf.compress_weights()``.
* Improved quantization time of LLMs with NNCF PTQ API for ``nncf.quantize()``
and ``nncf.quantize_with_accuracy_control()``.
* Added support for SmoothQuant and ChannelAlighnment algorithms in NNCF HyperParameter
Tuner for automatic optimization of their hyperparameters during quantization.
* Added quantization support for the ``IF`` operation of models in OpenVINO format
to speed up such models.
* NNCF Post-training Quantization for PyTorch backend is now supported with
``nncf.quantize()`` and the common implementation of quantization algorithms.
* Added support for PyTorch 2.1. PyTorch 1.13 support has been deprecated.
OpenVINO™ Runtime (previously known as Inference Engine)
---------------------------------------------------------
* Weight compression API, ``nncf.compress_weights()``, has been extended by:
* OpenVINO Common
* When using the 'all_layers' parameter, it compresses the model, including embeddings
and final layers, to the 4-bit format. This helps make the model footprint smaller
and improves performance, but it might impact the model accuracy. By default, this
parameter is disabled, and the backup precision (INT8) is assigned for the embeddings
and last layers.
* When using INT8_SYM compression mode for better performance of the compressed model
in case of 8-bit weight compression you might experience an impact on model accuracy
therefore by default, we use INT8_ASYM mode to better balance performance and accuracy.
* We implemented a 4-bit data-aware weight compression feature, introducing the 'dataset'
optional parameter in ``nncf.compress_weights()``. This parameter can be utilized to
mitigate accuracy loss in compressed models. It's important to note that enabling
this option will extend the compression time.
* Post-training Quantization with Accuracy Control, ``nncf.quantize_with_accuracy_control()``,
has been extended by the 'restore_mode' optional parameter to revert weights to INT8
instead of the original precision. This parameter helps to reduce the size of the
quantized model and improves its performance. By default, it is disabled and model
weights are reverted to the original precision in ``nncf.quantize_with_accuracy_control()``.
* Operations for reference implementations updated from legacy API to API 2.0.
* Symbolic transformation introduced the ability to remove Reshape operations
surrounding MatMul operations.
* OpenVINO Python API
* Better support for the ``openvino.properties`` submodule, which now allows the use
of properties directly, without additional parenthesis. Example use-case:
``{openvino.properties.cache_dir: “./some_path/”}``.
* Added missing properties: ``execution_devices`` and ``loaded_from_cache``.
* Improved error propagation on imports from OpenVINO package.
* AUTO device plug-in (AUTO)
* o Provided additional option to improve performance of cumulative throughput
(or MULTI), where part of CPU resources can be reserved for GPU inference
when GPU and CPU are both used for inference (using ``ov::hint::enable_cpu_pinning(true)``).
This avoids the performance issue of CPU resource contention where there
is not enough CPU resources to schedule tasks for GPU
(`PR #19214 <https://github.com/openvinotoolkit/openvino/pull/19214>`__).
* CPU
* Introduced support of GPTQ quantized Int4 models, with improved performance
compared to Int8 weight-compressed or FP16 models. In the CPU plugin,
the gain in performance is achieved by FullyConnected acceleration with
4bit weight decompression
(`PR #20607 <https://github.com/openvinotoolkit/openvino/pull/20607>`__).
* Improved performance of Int8 weight-compressed large language models on
some platforms, such as 13th Gen Intel Core
(`PR #20607 <https://github.com/openvinotoolkit/openvino/pull/20607>`__).
* Further reduced memory consumption of select large language models on
CPU platforms with AMX and AVX512 ISA, by eliminating extra memory copy
with a unified weight layout
(`PR #19575 <https://github.com/openvinotoolkit/openvino/pull/19575>`__).
* Fixed performance issue observed in 2023.1 release on select Xeon CPU
platform with improved thread workload partitioning matching L2 cache
utilization
(`PR #20436 <https://github.com/openvinotoolkit/openvino/pull/20436>`__).
* Extended support of configuration (enable_cpu_pinning) on Windows
platforms to allow fine-grain control on CPU resource used for inference
workload, by binding inference thread to CPU cores
(`PR #19418 <https://github.com/openvinotoolkit/openvino/pull/19418>`__).
* Optimized YoloV8n and YoloV8s model performance for BF16/FP32 precision.
* Optimized Falcon model on 4th Gen Intel® Xeon® Scalable Processors.
* Enabled support for FP16 inference precision on ARM.
* GPU
* Enhanced inference performance for Large Language Models.
* Introduced int8 weight compression to boost LLM performance.
(`PR #19548 <https://github.com/openvinotoolkit/openvino/pull/19548>`__).
* Implemented Int4 GPTQ weight compression for improved LLM performance.
* Optimized constant weights for LLMs, resulting in better memory usage
and faster model loading.
* Optimized gemm (general matrix multiply) and fc (fully connected) for
enhanced performance on iGPU.
(`PR #19780 <https://github.com/openvinotoolkit/openvino/pull/19780>`__).
* Completed GPU plugin migration to API 2.0.
* Added support for oneDNN 3.3 version.
OpenVINO™ Runtime
++++++++++++++++++++++++
* Model Import Updates
* TensorFlow Framework Support
* TensorFlow Framework Support
* Supported conversion of models from memory in keras.Model and tf.function formats.
`PR #19903 <https://github.com/openvinotoolkit/openvino/pull/19903>`__
* Supported TF 2.14.
`PR #20385 <https://github.com/openvinotoolkit/openvino/pull/20385>`__
* Supported TF1 While Control flow construction w/o TensorArray operations
(`PR #20800 <https://github.com/openvinotoolkit/openvino/pull/20800>`__).
* Support for complex tensors has been added
(`PR #20860 <https://github.com/openvinotoolkit/openvino/pull/20860>`__),
(`PR #21477 <https://github.com/openvinotoolkit/openvino/pull/21477>`__).
* Provided fixes for the following:
* PyTorch Framework Support
* Accept any model file extension for frozen protobuf format
(`PR #21508 <https://github.com/openvinotoolkit/openvino/pull/21508>`__).
* Correct ArgMin/ArgMax translators for repeating elements case
(`PR #21364 <https://github.com/openvinotoolkit/openvino/pull/21364>`__).
* Correct PartitionedCall translator when numbers of external and internal
body inputs mismatch
(`PR #20825 <https://github.com/openvinotoolkit/openvino/pull/20825>`__).
* Supported Int4 GPTQ models.
* New operations supported.
* PyTorch Framework Support
* ONNX Framework Support
* Added support of nested dictionaries and lists as example input.
* Disabled ``torch.jit.freeze`` in default model tracing scenario and
improved support for models without freezing, extending model
coverage and improving accuracy for some models.
* Added support for ONNX version 1.14.1
(`PR #18359 <https://github.com/openvinotoolkit/openvino/pull/18359>`__)
* ONNX Framework Support
* Switched to ONNX 1.15.0 as a supported version of original framework
(`PR #20929 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* CPU
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids)
with sub-numa (SNC) and efficient core resource scheduling to improve performance.
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Further improved performance of LLMs in INT4 weight compression, especially on 1st
token latency and on 4th and 5th Gen of Intel Xeon platforms (codename Sapphire
Rapids and Emerald Rapids) with AMX capabilities.
* Improved performance of transformer-based LLM using stateful model technique to
increase memory efficiency where internal states (KV cache) are shared among multiple
iterations of inference. The stateful model implementation supports both greedy search
and beam search (preview) for LLMs. This technique also reduces the memory footprint
of LLMs, where Intel Core and Ultra platforms like Raptor Lake and Meteor Lake can
run INT4 models, such as Llama v2 7B.
* Improved performance on ARM platforms with throughput hint, by increasing
efficiency in usage of the CPU cores and memory bandwidth.
* GPU
* Full support for Intel® Core™ Ultra (codename Meteor Lake) integrated graphics.
* For LLMs, the first inference latency for INT8 and INT4 weight-compressed models has
been improved on iGPU thanks to more efficient context processing. Overall average
token latency for INT8 and INT4 has also been enhanced on iGPU with graph compilation
optimization, various host overhead optimization, and dynamic padding support for GEMM.
* Stateful model is functionally supported for LLMs.
* Model caching for dynamically shaped models is now supported. Model loading time is
improved for these models, including LLMs.
* API for switching between size mode (model caching) and speed mode (kernel caching)
is introduced.
* The model cache file name is changed to be independent of GPU driver versions.
The GPU will not generate separate model cache files when the driver is updated.
* Compilation time for Stable Diffusion models has been improved.
* NPU
* NPU plugin is available as part of OpenVINO. With the Intel(R) Core Ultra NPU driver
installed, inference can run on the NPU device.
* AUTO device plug-in (AUTO)
* Introduced the round-robin policy to AUTO cumulative throughput hint, which dispatches
inference requests to multiple devices (such as multiple GPU devices) in the round-robin
sequence, instead of in the device priority sequence. The device priority sequence
remains as the default configuration.
* AUTO loads stateful models to GPU or CPU per device priority, since GPU now supports
stateful model inference.
* OpenVINO Common
* Enhanced support of String tensors has been implemented, enabling the use of operators
and models that rely on string tensors. This update also enhances the capability in
the torchvision preprocessing (`PR #21244 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* A new feature has been added that enables the selection of P-Cores for model compilation
on CPU device(s) with hybrid architecture (i.e. Intel® Core™ 12th Gen and beyond).
This will reduce compilation time compared to previous implementation where P-cores
and E-cores are used randomly by OS scheduling.
* OpenVINO JavaScript API (preview feature)
* We've introduced a preview version of
`JS API <https://github.com/openvinotoolkit/openvino/tree/master/src/bindings/js>`__
for OpenVINO runtime in this release. We hope that you will try this feature and
provide your feedback through GitHub `issues <https://github.com/openvinotoolkit/openvino/issues>`__.
* Known limitations:
* Only supported in manylinux and x86 (Windows, ARM, ARM64, and macOS have not been tested)
* Node.js version >= 18.16
* CMake version < 3.14 is not supported
* gcc compiler version < 7 is not supported
* OpenVINO Python API
* Introducing string tensor support for Python API.
* Added support for the following:
* Create ov.Tensor from Python lists
* Create ov.Tensor from empty numpy arrays.
* Constants from empty numpy arrays.
* Autogenerated get/set methods for Node attributes.
* Inference functions (InferRequest.infer/start_async, CompiledModel.__call__ etc.)
support OVDict as the input.
* PILLOW interpolation modes bindings.
(`PR #21188 <https://github.com/openvinotoolkit/openvino/pull/21188>`__
external contribution: @meetpatel0963)
* Torchvision to :doc:`OpenVINO preprocessing <openvino_docs_OV_UG_string_tensors>`
converter documentation has been added to OpenVINO docs.
OpenVINO Ecosystem
+++++++++++++++++++++++++++++++++++++++++++++
OpenVINO Model Server
--------------------------
* OpenVINO Tokenizer (Preview feature)
Introduced an extension of the KServe gRPC API, enabling streaming input and
output for servables with Mediapipe graphs. This extension ensures the persistence
of Mediapipe graphs within a user session, improving processing performance.
This enhancement supports stateful graphs, such as tracking algorithms, and
enables the use of source calculators.
(`see additional documentation <https://github.com/openvinotoolkit/model_server/blob/main/docs/streaming_endpoints.md>`__)
* OpenVINO Tokenizer adds text processing operations to OpenVINO:
* Mediapipe framework has been updated to the version 0.10.3.
* model_api used in the openvino inference Mediapipe calculator has been updated
and included with all its features.
* Added a demo showcasing gRPC streaming with Mediapipe graph.
(`see here <https://github.com/openvinotoolkit/model_server/tree/main/demos/mediapipe/holistic_tracking>`__)
* Added parameters for gRPC quota configuration and changed default gRPC channel
arguments to add rate limits. It will minimize the risks of impact of the service
from uncontrolled flow of requests.
* Updated python clients requirements to match wide range of python versions from 3.6 to 3.11
* Text PrePostprocessing without third-party dependencies
* Convert a HuggingFace tokenizer into the OpenVINO model tokenizer and the
detokenizer using a CLI tool or Python API
* Connect a tokenizer and a model to get a single model with text input
Learn more about the changes in https://github.com/openvinotoolkit/model_server/releases
Jupyter Notebook Tutorials
-----------------------------
* The following notebooks have been updated or newly added:
* `LaBSE <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/220-cross-lingual-books-alignment>`__
Cross-lingual Books Alignment With Transformers
* `LLM chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot>`__
Create LLM-powered Chatbot
* Updated to include Int4 weight compression and Zephyr 7B model
* `Bark Text-to-Speech <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/256-bark-text-to-audio>`__
Text-to-Speech generation using Bark
* `LLaVA Multimodal Chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/257-llava-multimodal-chatbot>`__
Visual-language assistant with LLaVA
* `BLIP-Diffusion - Subject-Driven Generation <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/258-blip-diffusion-subject-generation>`__
Subject-driven image generation and editing using BLIP Diffusion
* `DeciDiffusion <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/259-decidiffusion-image-generation>`__
Image generation with DeciDiffusion
* `Fast Segment Anything <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/261-fast-segment-anything>`__
Object segmentations with FastSAM
* `SoftVC VITS Singing Voice Conversion <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/262-softvc-voice-conversion>`__
* `QR Code Monster <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/264-qrcode-monster>`__
Generate creative QR codes with ControlNet QR Code Monster
* `Würstchen <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/265-wuerstchen-image-generation>`__
Text-to-image generation with Würstchen
* `Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/267-distil-whisper-asr>`__
Automatic speech recognition using Distil-Whisper and OpenVINO™
* OpenVINO Tokenizer models work only on the CPU device
* Supported platforms: Linux (x86 and ARM), Windows and Mac (x86 and ARM)
* Added optimization support (8-bit quantization, weight compression)
by NNCF for the following notebooks:
* OpenVINO Model Server
* Added support for serving pipelines with custom nodes implemented as a
`python code <https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/quickstart.md>`__
This greatly simplifies exposing GenAI algorithms based on Hugging Face
and Optimum libraries. It can be also applied for arbitrary pre and
post-processing in model serving pipelines.
* Included a new set of model serving demos that use custom nodes with python
code. These include LLM `text generation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__,
`stable diffusion <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/stable_diffusion>`__,
and `seq2seq translation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/seq2seq_translation>`__.
* Improved video stream analysis `demo <https://github.com/openvinotoolkit/model_server/tree/main/demos/real_time_stream_analysis/python>`__.
A simple client example can now process the
video stream from a local camera, video file or RTSP stream.
* Learn more about these changes on
`GitHub <https://github.com/openvinotoolkit/model_server/releases>`__.
* Jupyter Notebook Tutorials
* The following notebooks have been updated or newly added:
* `Sound generation with AudioLDM2 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/270-sound-generation-audioldm2>`__.
* `Single-step image generation using SDXL-turbo and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/271-sdxl-turbo>`__.
* `Paint by Example using Diffusion models and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/272-paint-by-example>`__.
* `LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/273-stable-zephyr-3b-chatbot>`__.
* `Object segmentations with EfficientSAM and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/274-efficient-sam>`__.
* `Create an LLM-powered RAG system using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-rag-chatbot.ipynb>`__
- Demonstrates an integration with LangChain.
* `High-resolution image generation with Segmind-VegaRT and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/248-stable-diffusion-xl/248-segmind-vegart.ipynb>`__.
* `Text-to-Image Generation with LCM LoRA and ControlNet Conditioning <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/263-latent-consistency-models-image-generation/263-lcm-lora-controlnet.ipynb>`__.
* `LLM Instruction-following pipeline with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/275-llm-question-answering>`__ -
Demonstrates how to run an instruction-following text generation pipeline using
tiny-llama-1b-chat, phi-2, dolly-v2-3b, red-pajama-3b-instruct and mistral-7b models.
* `LLM chabot notebook <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot>`__
updated with support for new LLMs and INT4/INT8 Weight Compression: TinyLlama-1b-chat,
Mistral-7B, neural-chat-7b, notus-7b, ChatGLM3, youri-7b-chat (for Japanese language).
* Added optimization support (8-bit quantization, weight compression) by NNCF for the following notebooks:
* `Image generation with Würstchen and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/265-wuerstchen-image-generation>`__
* `QR-code monster <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/264-qrcode-monster>`__
* `INT4-compression support for LLaVA multimodal chatbot <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/257-llava-multimodal-chatbot>`__
* `Distil-whisper quantization <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__
* `Image generation with DeepFloyd IF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/238-deepfloyd-if>`__
* `Instruction following using Databricks Dolly 2.0 <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/240-dolly-2-instruction-following>`__
* `Visual Question Answering and Image Captioning using BLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/233-blip-visual-language-processing>`__
* `Grammatical Error Correction <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/214-grammar-correction>`__
* `Universal segmentation with OneFormer <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/249-oneformer-segmentation>`__
* `Visual-language assistant with LLaVA and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/257-llava-multimodal-chatbot>`__
* `Image editing with InstructPix2Pix <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/231-instruct-pix2pix-image-editing>`__
* `MMS: Scaling Speech Technology to 1000+ languages <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/255-mms-massively-multilingual-speech>`__
* `Image generation with Latent Consistency Model <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/263-latent-consistency-models-image-generation>`__
* `Object segmentations with FastSAM <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/261-fast-segment-anything>`__
* `Automatic speech recognition using Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__
Known issues
++++++++++++++++++++++++++++++++++++++++++++
| **ID - 118179**
| *Component* - Python API, Plugins
| **ID - 127202**
| *Component* - CPU Plugin
| *Description:*
| When input byte sizes are matching, inference methods accept incorrect inputs
in copy mode (share_inputs=False). Example: [1, 4, 512, 512] is allowed when
[1, 512, 512, 4] is required by the model.
| Deeplabv3 model from TF framework shows lower performance than previous
release. This is because the TopK layer in the model is now correctly
conducting the stable sort as specified by the model, slower than the
previous unstable sort.
| *Workaround:*
| Pass inputs which shape and layout match model ones.
| **ID - 124181**
| *Component* - CPU plugin
| *Description:*
| On CPU platform with L2 cache size less than 256KB, such as i3 series of 8th
Gen Intel CORE platforms, some models may hang during model loading.
| *Workaround:*
| Rebuild the software from OpenVINO master or use the next OpenVINO release.
| **ID - 121959**
| *Component* - CPU plugin
| *Description:*
| During inference using latency hint on selected hybrid CPU platforms
(such as 12th or 13th Gen Intel CORE), there is a sporadic occurrence of
increased latency caused by the operating system scheduling of P-cores or
E-cores during OpenVINO initialization.
| *Workaround:*
| This will be fixed in the next OpenVINO release.
| This release has the correct behavior. If performance is critical,
please use the previous version of OpenVINO, or tune the model.
| **ID - 123101**
| *Component* - GPU plugin
| *Component* - GPU plugin
| *Description:*
| Hung up of GPU plugin on A770 Graphics (dGPU) in case of
large batch size (1750).
| Hung up of GPU plugin on A770 Graphics (dGPU) in case of large
batch size (1750).
| *Workaround:*
| Decrease the batch size, wait for fixed driver released.
| Decrease the batch size, and wait for the fixed driver released.
Included in This Release
+++++++++++++++++++++++++++++++++++++++++++++
@ -320,19 +353,19 @@ three types of operating systems: Windows, Linux, and macOS.
|| Component || License | Location |
+================================+===================================+=================+=================+=======================+=================================================+
|| OpenVINO (Inference Engine) C++ Runtime || Dual licensing: || <install_root>/runtime/* |
|| Unified API to integrate the inference with application logic || Intel® OpenVINO™ Distribution License (Version May 2021) || <install_root>/runtime/include/* |
|| OpenVINO (Inference Engine) Headers || Apache 2.0 || |
|| Unified API to integrate the inference with application logic || Intel® OpenVINO™ Distribution License (Version May 2021) || <install_root>/runtime/include/* |
|| OpenVINO (Inference Engine) Headers || Apache 2.0 || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Pythion API || Apache 2.0 || <install_root>/python/* |
|| OpenVINO (Inference Engine) Python API || Apache 2.0 || <install_root>/python/* |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Samples || Apache 2.0 || <install_root>/samples/* |
|| Samples that illustrate OpenVINO C++/ Python API usage || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| [Deprecated] Deployment manager || Apache 2.0 || <install_root>/tools/deployment_manager/* |
|| The Deployment Manager is a Python* command-line tool that || || |
|| creates a deployment package by assembling the model, IR files, || || |
|| your application, and associated dependencies into a runtime || || |
|| package for your target device. || || |
|| [Deprecated] Deployment manager || Apache 2.0 || <install_root>/tools/deployment_manager/* |
|| The Deployment Manager is a Python command-line tool that || || |
|| creates a deployment package by assembling the model, IR files, || || |
|| your application, and associated dependencies into a runtime || || |
|| package for your target device. || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
@ -360,7 +393,7 @@ enabled hardware, software or service activation. Learn more at
`http://www.intel.com/ <http://www.intel.com/>`__
or from the OEM or retailer.
No computer system can be absolutely secure.
No computer system can be absolutely secure.
Intel, Atom, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks
of Intel Corporation in the U.S. and/or other countries.
@ -371,18 +404,18 @@ Other names and brands may be claimed as the property of others.
Copyright © 2023, Intel Corporation. All rights reserved.
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
`www.Intel.com/PerformanceIndex <www.Intel.com/PerformanceIndex>`__.
Download
+++++++++++++++++++++++++++++++++++++++++++++
`The OpenVINO product selector tool <https://docs.openvino.ai/install>`__
provides easy access to the right packages that match your desired OS, version,
provides easy access to the right packages that match your desired OS, version,
and distribution options.

124
docs/articles_en/about_openvino/system_requirements.rst
View File

@ -2,13 +2,16 @@
System Requirements
===================
Certain hardware requires specific drivers to work properly with OpenVINO.
These drivers, including Linux* kernels, might require updates to your operating system,
which is not part of OpenVINO installation. Refer to your hardware's documentation
for updating instructions.
.. note::
Certain hardware (including but not limited to GPU, GNA, and latest CPUs) requires manual
installation of specific drivers and/or other software components to work correctly and/or
to utilize hardware capabilities at their best. This might require updates to operating
system, including but not limited to Linux kernel, please refer to their documentation
for details. These modifications should be handled by user and are not part of OpenVINO
installation.
CPU
@ -20,26 +23,24 @@ CPU
* Intel Atom® processor with Intel® SSE4.2 support
* Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
* 6th - 13th generation Intel® Core™ processors
* Intel® Core™ Ultra (codename Meteor Lake)
* Intel® Xeon® Scalable Processors (code name Skylake)
* 2nd Generation Intel® Xeon® Scalable Processors (code name Cascade Lake)
* 3rd Generation Intel® Xeon® Scalable Processors (code name Cooper Lake and Ice Lake)
* 4th Generation Intel® Xeon® Scalable Processors (code name Sapphire Rapids)
* ARM* and ARM64 CPUs; Apple M1, M2 and Raspberry Pi
* 6th - 14th generation Intel® Core™ processors
* Intel® Core™ Ultra (codename Meteor Lake)
* 1st - 5th generation Intel® Xeon® Scalable Processors
* ARM and ARM64 CPUs; Apple M1, M2, and Raspberry Pi
.. tab-item:: Supported Operating Systems
* Ubuntu 22.04 long-term support (LTS), 64-bit (Kernel 5.15+)
* Ubuntu 20.04 long-term support (LTS), 64-bit (Kernel 5.15+)
* Ubuntu 18.04 long-term support (LTS) with limitations, 64-bit (Kernel 5.4+), ARM64
* Windows* 10
* Windows* 11
* macOS* 10.15 and above, 64-bit
* Ubuntu 18.04 long-term support (LTS) with limitations, 64-bit (Kernel 5.4+)
* Windows 10
* Windows 11
* macOS 10.15 and above, 64-bit
* macOS 11 and above, ARM64
* Red Hat Enterprise Linux* 8, 64-bit
* CentOS 7
* Red Hat Enterprise Linux 8, 64-bit
* Ubuntu 18 ARM64
* Debian 9 ARM
* CentOS 7 64-bit
GPU
##########
@ -53,7 +54,7 @@ GPU
* Intel® Iris® Pro Graphics
* Intel® Iris® Xe Graphics
* Intel® Iris® Xe Max Graphics
* Intel® Arc ™ GPU Series
* Intel® Arc™ GPU Series
* Intel® Data Center GPU Flex Series
* Intel® Data Center GPU Max Series
@ -63,7 +64,7 @@ GPU
* Ubuntu 20.04 long-term support (LTS), 64-bit
* Windows 10, 64-bit
* Windows 11, 64-bit
* Centos 7
* CentOS 7
* Red Hat Enterprise Linux 8, 64-bit
.. tab-item:: Additional considerations
@ -72,17 +73,17 @@ GPU
Distribution of OpenVINO™ toolkit package.
* A chipset that supports processor graphics is required for Intel® Xeon®
processors. Processor graphics are not included in all processors. See
`Product Specifications <https://ark.intel.com/>`__
for information about your processor.
* Although this release works with Ubuntu 20.04 for discrete graphic cards,
Ubuntu 20.04 is not POR for discrete graphics drivers, so OpenVINO support
is limited.
`Product Specifications <https://ark.intel.com/>`__
for information about your processor.
* While this release of OpenVINO supports Ubuntu 20.04, the driver stack
for Intel discrete graphic cards does not fully support Ubuntu 20.04.
We recommend using Ubuntu 22.04 when executing on discrete graphics.
* The following minimum (i.e., used for old hardware) OpenCL™ driver's versions
were used during OpenVINO internal validation: 22.43 for Ubuntu 22.04, 21.48
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8.
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8.
NPU and GNA
#############################
Intel® Neural Processing Unit
################################
.. tab-set::
@ -100,10 +101,23 @@ NPU and GNA
.. tab-item:: Additional considerations
* These Accelerators require drivers that are not included in the
Intel® Distribution of OpenVINO™ toolkit package.
* These Accelerators require :doc:`drivers <openvino_docs_install_guides_configurations_for_intel_npu>`
that are not included in the Intel® Distribution of OpenVINO™ toolkit package.
* Users can access the NPU plugin through the OpenVINO archives on
the download page.
the :doc:`download page <openvino_docs_install_guides_overview>`.
Intel® Gaussian & Neural Accelerator
##########################################
.. tab-set::
.. tab-item:: Operating Systems for GNA
* Ubuntu 22.04 long-term support (LTS), 64-bit
* Ubuntu 20.04 long-term support (LTS), 64-bit
* Windows 10, 64-bit
* Windows 11, 64-bit
Operating systems and developer environment
@ -111,7 +125,7 @@ Operating systems and developer environment
.. tab-set::
.. tab-item:: Linux
.. tab-item:: Linux OS
* Ubuntu 22.04 with Linux kernel 5.15+
* Ubuntu 20.04 with Linux kernel 5.15+
@ -127,16 +141,13 @@ Operating systems and developer environment
* `GNU Compiler Collection (GCC) <https://www.gnu.org/software/gcc/>`__ 7.5 and above
* `CMake <https://cmake.org/download/>`__ 3.10 or higher
Higher versions of kernel might be required for 10th Gen Intel® Core™ Processors,
11th Gen Intel® Core™ Processors, 11th Gen Intel® Core™ Processors S-Series Processors,
12th Gen Intel® Core™ Processors, 13th Gen Intel® Core™ Processors, Intel® Core™ Ultra
Processors, or 4th Gen Intel® Xeon® Scalable Processors to support CPU, GPU, GNA or
hybrid-cores CPU capabilities.
Higher versions of kernel might be required for 10th Gen Intel® Core™ Processors, 11th Gen
Intel® Core™ Processors, 11th Gen Intel® Core™ Processors S-Series Processors, 12th Gen
Intel® Core™ Processors, 13th Gen Intel® Core™ Processors, 14th Gen
Intel® Core™ Processors, Intel® Core™ Ultra Processors, 4th Gen Intel® Xeon® Scalable Processors
or 5th Gen Intel® Xeon® Scalable Processors to support CPU, GPU, GNA or hybrid-cores CPU capabilities.
.. tab-item:: Windows
* Windows 10
* Windows 11
.. tab-item:: Windows 10 and 11
Build environment components:
@ -158,26 +169,25 @@ Operating systems and developer environment
.. tab-item:: DL frameworks versions:
* TensorFlow* 1.15, 2.12
* MxNet* 1.9.0
* ONNX* 1.14.1
* PaddlePaddle* 2.4
* TensorFlow 1.15, 2.12
* MxNet 1.9.0
* ONNX 1.14.1
* PaddlePaddle 2.4
This package can be installed on other versions of DL Framework
but only the version specified here is fully validated.
This package can be installed on other versions of DL Frameworks
but only the versions specified here are fully validated.
.. note::
OpenVINO Python binaries and binaries on Windows/CentOS7/MACOS(x86) are built
with oneTBB libraries. Other binaries on Ubuntu and Redhat OSes are built with
legacy TBB which is released by OS distribution. OpenVINO can be built with
either oneTBB or legacy TBB by the user on all OS systems listed. System
compatibility and performance are improved on Hybrid CPUs,
OpenVINO Python binaries and binaries on Windows, CentOS 7, and macOS (x86) are built
with oneTBB libraries, and others on Ubuntu and RedHat systems are built with
legacy TBB which is released by OS distribution. OpenVINO can be built from source
with either oneTBB or legacy TBB on all the systems listed here. System
compatibility and performance are improved on Hybrid CPUs
such as 12th Gen Intel Core and above.
Legal Information
+++++++++++++++++++++++++++++++++++++++++++++
@ -202,7 +212,7 @@ enabled hardware, software or service activation. Learn more at
`http://www.intel.com/ <http://www.intel.com/>`__
or from the OEM or retailer.
No computer system can be absolutely secure.
No computer system can be absolutely secure.
Intel, Atom, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks
of Intel Corporation in the U.S. and/or other countries.
@ -213,9 +223,9 @@ Other names and brands may be claimed as the property of others.
Copyright © 2023, Intel Corporation. All rights reserved.
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
`www.Intel.com/PerformanceIndex <www.Intel.com/PerformanceIndex>`__.