natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge 4ca9fa58b6 into 23fc29a6db

This commit is contained in:
Tatiana Savina 2024-06-12 08:06:12 +02:00 committed by GitHub
parent 23fc29a6db 4ca9fa58b6
commit 4993d29d55
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 384 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
docs/articles_en/get-started/install-openvino.rst
View File

@ -14,7 +14,7 @@ Install OpenVINO™ 2024.2
OpenVINO Runtime on Linux <install-openvino/install-openvino-linux>
OpenVINO Runtime on Windows <install-openvino/install-openvino-windows>
OpenVINO Runtime on macOS <install-openvino/install-openvino-macos>
OpenVINO GenAI Flavor <install-openvino/install-openvino-genai>
.. raw:: html
@ -22,17 +22,7 @@ Install OpenVINO™ 2024.2
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<iframe id="selector" src="../_static/selector-tool/selector-1c16038.html" style="width: 100%; border: none" title="Download Intel® Distribution of OpenVINO™ Toolkit"></iframe>
.. warning::
The OpenVINO™ Development Tools package has been deprecated and removed from the default
installation options. For new projects, the OpenVINO runtime package now includes
all necessary components.
The OpenVINO Development Tools is still available for older versions of OpenVINO,
as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`.
.. tip::
.. dropdown:: Supported OpenVINO Versions
OpenVINO 2024.2, described here, is not a Long-Term-Support version!
All currently supported versions are:
@ -43,6 +33,8 @@ Install OpenVINO™ 2024.2
Moreover, different OpenVINO distributions may support slightly different sets of features.
Read installation guides for particular distributions for more details.
Refer to the :doc:`OpenVINO Release Policy <../../../about-openvino/release-notes-openvino/release-policy>`
to learn more about the release types.
.. dropdown:: Distribution Comparison for OpenVINO 2024.2
@ -57,9 +49,27 @@ Install OpenVINO™ 2024.2
| \* **Of the Linux systems, versions 22.04 and 24.04 include drivers for NPU.**
| **For Windows, CPU inference on ARM64 is not supported.**
| **Build OpenVINO from source**
| OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
follow the `OpenVINO Build Instructions <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>`__.
.. dropdown:: Effortless GenAI integration with OpenVINO GenAI Flavor
A new OpenVINO GenAI Flavor streamlines application development by providing
LLM-specific interfaces for easy integration of language models, handling tokenization and
text generation. For installation and usage instructions, proceed to
:doc:`Install OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>` and
:doc:`Run LLMs with OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>`.
.. dropdown:: Deprecation of OpenVINO™ Development Tools Package
The OpenVINO™ Development Tools package has been deprecated and removed from the default
installation options. For new projects, the OpenVINO runtime package now includes
all necessary components.
The OpenVINO Development Tools is still available for older versions of OpenVINO,
as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`.
.. dropdown:: Building OpenVINO from source
OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
follow the `OpenVINO Build Instructions <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>`__.

3
docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst
View File

@ -32,7 +32,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File
RHEL8 x86_64 V V n/a
=================== ===== ===== =====
.. tab-set::
.. tab-item:: System Requirements
@ -94,8 +93,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File
Installing OpenVINO Runtime
############################################################

62
docs/articles_en/get-started/install-openvino/install-openvino-genai.rst Normal file
View File

@ -0,0 +1,62 @@
Install OpenVINO™ GenAI
====================================
OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of generative AI models.
It hides the complexity of the generation process and minimizes the amount of code required.
You can now provide a model and input context directly to OpenVINO, which performs tokenization of the
input text, executes the generation loop on the selected device, and returns the generated text.
For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`.
To see GenAI in action, check the Jupyter notebooks:
`LLM-powered Chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/README.md>`__ and
`LLM Instruction-following pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-question-answering/README.md>`__
The OpenVINO GenAI flavor is available for installation via Archive and PyPI distributions:
Archive Installation
###############################
To install the GenAI flavor of OpenVINO from an archive file, follow the standard installation steps for your system
but instead of using the vanilla package file, download the one with OpenVINO GenAI:
.. tab-set::
.. tab-item:: x86_64
:sync: x86-64
.. tab-set::
.. tab-item:: Ubuntu 24.04
:sync: ubuntu-24
.. code-block:: sh
curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz
tar -xf openvino_2024.1.0.tgz
sudo mv l_openvino_toolkit_ubuntu24_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0
.. tab-item:: Ubuntu 22.04
:sync: ubuntu-22
.. code-block:: sh
curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz
tar -xf openvino_2024.1.0.tgz
sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0
Here are the full guides:
:doc:`Linux <install-openvino-archive-linux>`,
:doc:`Windows <install-openvino-archive-windows>`, and
:doc:`macOS <install-openvino-archive-macos>`.
PyPI Installation
###############################
To install the GenAI flavor of OpenVINO via PyPI, follow the standard :doc:`installation steps <install-openvino-pip>`,
but use the *openvino-genai* package instead of *openvino*:
.. code-block:: python
python -m pip install openvino-genai

1
docs/articles_en/get-started/install-openvino/install-openvino-linux.rst
View File

@ -26,7 +26,6 @@ Install OpenVINO™ Runtime on Linux
Use Docker <install-openvino-docker-linux>
Use npm <install-openvino-npm>
If you want to install OpenVINO™ Runtime on Linux, you have the following options:
* :doc:`Install OpenVINO using an Archive File <install-openvino-archive-linux>`

5
docs/articles_en/get-started/install-openvino/install-openvino-pip.rst
View File

@ -100,13 +100,12 @@ Use the following command:
Step 4. Install the Package
+++++++++++++++++++++++++++
Use the following command:
Use the following command to install OpenVINO Base or OpenVINO GenAI package:
.. code-block:: sh
.. code-block:: python
python -m pip install openvino
Step 5. Verify that the Package Is Installed
++++++++++++++++++++++++++++++++++++++++++++

10
docs/articles_en/learn-openvino/llm_inference_guide.rst
View File

@ -13,8 +13,9 @@ Large Language Model Inference Guide
:maxdepth: 1
:hidden:
LLM Inference with Optimum Intel <llm_inference_guide/llm-inference-hf>
LLM Inference with OpenVINO API <llm_inference_guide/llm-inference-native-ov>
Run LLMs with Optimum Intel <llm_inference_guide/llm-inference-hf>
Run LLMs with OpenVINO GenAI Flavor <llm_inference_guide/genai-guide>
Run LLMs with Base OpenVINO <llm_inference_guide/llm-inference-native-ov>
OpenVINO Tokenizers <llm_inference_guide/ov-tokenizers>
Large Language Models (LLMs) like GPT are transformative deep learning networks capable of a
@ -39,12 +40,13 @@ The advantages of using OpenVINO for LLM deployment:
* **Provides stateful model optimization**: models from the Hugging Face Transformers are converted into a stateful form, optimizing inference performance and memory usage in long-running text generation tasks by managing past KV-cache tensors more efficiently internally. This feature is automatically activated for many supported models, while unsupported ones remain stateless. Learn more about the :doc:`Stateful models and State API <../openvino-workflow/running-inference/stateful-models>`.
OpenVINO offers two main paths for Generative AI use cases:
OpenVINO offers three main paths for Generative AI use cases:
* **OpenVINO GenAI Flavor**: use OpenVINO GenAI APIs (Python and C++).
* **Hugging Face**: use OpenVINO as a backend for Hugging Face frameworks (transformers,
diffusers) through the `Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__
extension.
* **Native OpenVINO**: use OpenVINO native APIs (Python and C++) with
* **Base OpenVINO**: use OpenVINO native APIs (Python and C++) with
`custom pipeline code <https://github.com/openvinotoolkit/openvino.genai>`__.
In both cases, the OpenVINO runtime is used for inference, and OpenVINO tools are used for

287
docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst Normal file
View File

@ -0,0 +1,287 @@
Run LLMs with OpenVINO GenAI Flavor
=====================================
.. meta::
:description: Learn how to use the OpenVINO GenAI flavor to execute LLM models.
This guide will show you how to integrate the OpenVINO GenAI flavor into your application, covering
loading a model and passing the input context to receive generated text. Note that the vanilla flavor of OpenVINO
will not work with these instructions, make sure to
:doc:`install OpenVINO GenAI <../../get-started/install-openvino/install-openvino-genai>`.
.. note::
The examples use the CPU as the target device, however, the GPU is also supported.
Note that for the LLM pipeline, the GPU is used only for inference, while token selection, tokenization, and
detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run
on the CPU.
1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example:
.. code-block:: python
optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"
*Optional*. Optimize the model:
The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance,
it is recommended to use lower precision for model weights, such as INT4, and to compress weights
using NNCF during model export directly:
.. code-block:: python
optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code
2. Perform generation using the new GenAI API:
.. tab-set::
.. tab-item:: Python
:sync: py
.. code-block:: python
import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(model_path, "CPU")
print(pipe.generate("The Sun is yellow because"))
.. tab-item:: C++
:sync: cpp
.. code-block:: cpp
#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>
int main(int argc, char* argv[]) {
std::string model_path = argv[1];
ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU
std::cout << pipe.generate("The Sun is yellow because"); //input context
The `LLMPipeline` is the main object used for decoding. You can construct it directly from the
folder with the converted model. It will automatically load the main model, tokenizer, detokenizer,
and the default generation configuration.
Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information
necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that
its results match those generated by Hugging Face.
Streaming the Output
###########################
For more interactive UIs during generation, streaming of model output tokens is supported. See the example
below, where a lambda function outputs words to the console immediately upon generation:
.. tab-set::
.. tab-item:: C++
.. code-block:: cpp
#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>
int main(int argc, char* argv[]) {
std::string model_path = argv[1];
ov::genai::LLMPipeline pipe(model_path, "CPU");
auto streamer = [](std::string word) { std::cout << word << std::flush; };
std::cout << pipe.generate("The Sun is yellow because", streamer);
}
You can also create your custom streamer for more sophisticated processing:
.. tab-set::
.. tab-item:: C++
.. code-block:: cpp
#include <streamer_base.hpp>
class CustomStreamer: publict StreamerBase {
public:
void put(int64_t token) {/* decode tokens and do process them*/};
void end() {/* decode tokens and do process them*/};
};
int main(int argc, char* argv[]) {
CustomStreamer custom_streamer;
std::string model_path = argv[1];
ov::LLMPipeline pipe(model_path, "CPU");
cout << pipe.generate("The Sun is yellow because", custom_streamer);
}
Optimizing the Chat Scenario
################################
For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs
may prove beneficial. The chat-specific methods **start_chat** and **finish_chat** are used to
mark a conversation session, as you can see in these simple examples:
.. tab-set::
.. tab-item:: Python
:sync: py
.. code-block:: python
import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(model_path)
config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1}
pipe.set_generation_cofnig(config)
pipe.start_chat()
while True:
    print('question:')
    prompt = input()
if prompt == 'Stop!':
        break
    print(pipe(prompt))
pipe.finish_chat()
.. tab-item:: C++
:sync: cpp
.. code-block:: cpp
int main(int argc, char* argv[]) {
std::string prompt;
std::string model_path = argv[1];
ov::LLMPipeline pipe(model_path, "CPU");
pipe.start_chat();
for (size_t i = 0; i < questions.size(); i++) {
std::cout << "question:\n";
std::getline(std::cin, prompt);
std::cout << pipe(prompt) << std::endl>>;
}
pipe.finish_chat();
}
Optimizing Generation with Grouped Beam Search
#######################################################
Leverage grouped beam search decoding and configure generation_config for better text generation
quality and efficient batch processing in GenAI applications.
Use grouped beam search decoding:
.. tab-set::
.. tab-item:: C++
.. code-block:: cpp
int main(int argc, char* argv[]) {
std::string model_path = argv[1];
ov::LLMPipeline pipe(model_path, "CPU");
ov::GenerationConfig config = pipe.get_generation_config();
config.max_new_tokens = 256;
config.num_groups = 3;
config.group_size = 5;
config.diversity_penalty = 1.0f;
cout << pipe.generate("The Sun is yellow because", config);
}
Specify generation_config to use grouped beam search:
.. tab-set::
.. tab-item:: C++
.. code-block:: cpp
int main(int argc, char* argv[]) {
std::string prompt;
std::string model_path = argv[1];
ov::LLMPipeline pipe(model_path, "CPU");
ov::GenerationConfig config = pipe.get_generation_config();
config.max_new_tokens = 256;
config.num_groups = 3;
config.group_size = 5;
config.diversity_penalty = 1.0f;
auto streamer = [](std::string word) { std::cout << word << std::flush; };
pipe.start_chat();
for (size_t i = 0; i < questions.size(); i++) {
std::cout << "question:\n";
cout << prompt << endl;
auto answer = pipe(prompt, config, streamer);
// no need to print answer, streamer will do that
}
pipe.finish_chat();
}
Comparing with Hugging Face Results
#######################################
Compare and analyze results with those generated by Hugging Face models.
.. tab-set::
.. tab-item:: Python
.. code-block:: python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
max_new_tokens = 32
prompt = 'table is made of'
encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False)
hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False)
hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:])
print(f'hf_output: {hf_output}')
import sys
sys.path.append('build-Debug/')
import py_generate_pipeline as genai # set more friendly module name
pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/')
ov_output = pipe(prompt, max_new_tokens=max_new_tokens)
print(f'ov_output: {ov_output}')
assert hf_output == ov_output
GenAI API
#######################################
OpenVINO GenAI Flavor includes the following API:
* generation_config - defines a configuration class for text generation, enabling customization of the generation process such as the maximum length of the generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding strategy (greedy, beam search, or multinomial sampling).
* llm_pipeline - provides classes and utilities for text generation, including a pipeline for processing inputs, generating text, and managing outputs with configurable options.
* streamer_base - an abstract base class for creating streamers.
* tokenizer - the tokenizer class for text encoding and decoding.
* visibility - controls the visibility of the GenAI library.
Learn more about API in the `GenAI repository <https://github.com/openvinotoolkit/openvino.genai/tree/master/src/cpp/include/openvino/genai>`__.
Additional Resources
####################
* `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__
* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__

2
docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst
View File

@ -1,6 +1,6 @@
.. {#llm_inference}
Inference with Hugging Face and Optimum Intel
Run LLMs with Hugging Face and Optimum Intel
=====================================================
The steps below show how to load and infer LLMs from Hugging Face using Optimum Intel.

4
docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst
View File

@ -1,6 +1,6 @@
.. {#llm_inference_native_ov}
Inference with Native OpenVINO
Run LLMs with Base OpenVINO
===============================
To run Generative AI models using native OpenVINO APIs you need to follow regular **Convert -> Optimize -> Deploy** path with a few simplifications.
@ -21,7 +21,6 @@ Inference code that uses native API cannot benefit from Hugging Face pipelines.
To write such pipelines, you can follow the examples provided as part of OpenVINO:
* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
* `OpenVINO Latent Consistency Model C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/lcm_dreamshaper_v7/cpp>`__
* `OpenVINO Stable Diffusion (with LoRA) C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/stable_diffusion_1_5/cpp>`__
@ -174,7 +173,6 @@ This step is essential for interpreting the model's output.
Additional Resources
####################
* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
* `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__
* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__