diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index 70c53b53891..6c6eac4c592 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -14,7 +14,7 @@ Install OpenVINO™ 2024.2 OpenVINO Runtime on Linux OpenVINO Runtime on Windows OpenVINO Runtime on macOS - + OpenVINO GenAI Flavor .. raw:: html @@ -22,17 +22,7 @@ Install OpenVINO™ 2024.2 -.. warning:: - - The OpenVINO™ Development Tools package has been deprecated and removed from the default - installation options. For new projects, the OpenVINO runtime package now includes - all necessary components. - - The OpenVINO Development Tools is still available for older versions of OpenVINO, - as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`. - - -.. tip:: +.. dropdown:: Supported OpenVINO Versions OpenVINO 2024.2, described here, is not a Long-Term-Support version! All currently supported versions are: @@ -43,6 +33,8 @@ Install OpenVINO™ 2024.2 Moreover, different OpenVINO distributions may support slightly different sets of features. Read installation guides for particular distributions for more details. + Refer to the :doc:`OpenVINO Release Policy <../../../about-openvino/release-notes-openvino/release-policy>` + to learn more about the release types. .. dropdown:: Distribution Comparison for OpenVINO 2024.2 @@ -57,9 +49,27 @@ Install OpenVINO™ 2024.2 | \* **Of the Linux systems, versions 22.04 and 24.04 include drivers for NPU.** | **For Windows, CPU inference on ARM64 is not supported.** -| **Build OpenVINO from source** -| OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform, - follow the `OpenVINO Build Instructions `__. +.. dropdown:: Effortless GenAI integration with OpenVINO GenAI Flavor + + A new OpenVINO GenAI Flavor streamlines application development by providing + LLM-specific interfaces for easy integration of language models, handling tokenization and + text generation. For installation and usage instructions, proceed to + :doc:`Install OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>` and + :doc:`Run LLMs with OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>`. + +.. dropdown:: Deprecation of OpenVINO™ Development Tools Package + + The OpenVINO™ Development Tools package has been deprecated and removed from the default + installation options. For new projects, the OpenVINO runtime package now includes + all necessary components. + + The OpenVINO Development Tools is still available for older versions of OpenVINO, + as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`. + +.. dropdown:: Building OpenVINO from source + + OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform, + follow the `OpenVINO Build Instructions `__. diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst index 32b7b36ce5b..a991bae6fe0 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst @@ -32,7 +32,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File RHEL8 x86_64 V V n/a =================== ===== ===== ===== - .. tab-set:: .. tab-item:: System Requirements @@ -94,8 +93,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File - - Installing OpenVINO Runtime ############################################################ diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst new file mode 100644 index 00000000000..5927d11b18e --- /dev/null +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -0,0 +1,62 @@ +Install OpenVINO™ GenAI +==================================== + +OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of generative AI models. +It hides the complexity of the generation process and minimizes the amount of code required. +You can now provide a model and input context directly to OpenVINO, which performs tokenization of the +input text, executes the generation loop on the selected device, and returns the generated text. +For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`. + +To see GenAI in action, check the Jupyter notebooks: +`LLM-powered Chatbot `__ and +`LLM Instruction-following pipeline `__ + +The OpenVINO GenAI flavor is available for installation via Archive and PyPI distributions: + +Archive Installation +############################### + +To install the GenAI flavor of OpenVINO from an archive file, follow the standard installation steps for your system +but instead of using the vanilla package file, download the one with OpenVINO GenAI: + +.. tab-set:: + + .. tab-item:: x86_64 + :sync: x86-64 + + .. tab-set:: + + .. tab-item:: Ubuntu 24.04 + :sync: ubuntu-24 + + .. code-block:: sh + + + curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz + tar -xf openvino_2024.1.0.tgz + sudo mv l_openvino_toolkit_ubuntu24_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 + + .. tab-item:: Ubuntu 22.04 + :sync: ubuntu-22 + + .. code-block:: sh + + curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz + tar -xf openvino_2024.1.0.tgz + sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 + +Here are the full guides: +:doc:`Linux `, +:doc:`Windows `, and +:doc:`macOS `. + +PyPI Installation +############################### + +To install the GenAI flavor of OpenVINO via PyPI, follow the standard :doc:`installation steps `, +but use the *openvino-genai* package instead of *openvino*: + +.. code-block:: python + + python -m pip install openvino-genai + diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst index 38561aaebae..36df7f39ff8 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst @@ -26,7 +26,6 @@ Install OpenVINO™ Runtime on Linux Use Docker Use npm - If you want to install OpenVINO™ Runtime on Linux, you have the following options: * :doc:`Install OpenVINO using an Archive File ` diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst index ed3c1b12428..191d7563f6d 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst @@ -100,13 +100,12 @@ Use the following command: Step 4. Install the Package +++++++++++++++++++++++++++ -Use the following command: +Use the following command to install OpenVINO Base or OpenVINO GenAI package: -.. code-block:: sh +.. code-block:: python python -m pip install openvino - Step 5. Verify that the Package Is Installed ++++++++++++++++++++++++++++++++++++++++++++ diff --git a/docs/articles_en/learn-openvino/llm_inference_guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide.rst index b26c3adc74d..bb8ae80ee3b 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide.rst @@ -13,8 +13,9 @@ Large Language Model Inference Guide :maxdepth: 1 :hidden: - LLM Inference with Optimum Intel - LLM Inference with OpenVINO API + Run LLMs with Optimum Intel + Run LLMs with OpenVINO GenAI Flavor + Run LLMs with Base OpenVINO OpenVINO Tokenizers Large Language Models (LLMs) like GPT are transformative deep learning networks capable of a @@ -39,12 +40,13 @@ The advantages of using OpenVINO for LLM deployment: * **Provides stateful model optimization**: models from the Hugging Face Transformers are converted into a stateful form, optimizing inference performance and memory usage in long-running text generation tasks by managing past KV-cache tensors more efficiently internally. This feature is automatically activated for many supported models, while unsupported ones remain stateless. Learn more about the :doc:`Stateful models and State API <../openvino-workflow/running-inference/stateful-models>`. -OpenVINO offers two main paths for Generative AI use cases: +OpenVINO offers three main paths for Generative AI use cases: +* **OpenVINO GenAI Flavor**: use OpenVINO GenAI APIs (Python and C++). * **Hugging Face**: use OpenVINO as a backend for Hugging Face frameworks (transformers, diffusers) through the `Optimum Intel `__ extension. -* **Native OpenVINO**: use OpenVINO native APIs (Python and C++) with +* **Base OpenVINO**: use OpenVINO native APIs (Python and C++) with `custom pipeline code `__. In both cases, the OpenVINO runtime is used for inference, and OpenVINO tools are used for diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst new file mode 100644 index 00000000000..b3088dc3aa7 --- /dev/null +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -0,0 +1,287 @@ +Run LLMs with OpenVINO GenAI Flavor +===================================== + +.. meta:: + :description: Learn how to use the OpenVINO GenAI flavor to execute LLM models. + +This guide will show you how to integrate the OpenVINO GenAI flavor into your application, covering +loading a model and passing the input context to receive generated text. Note that the vanilla flavor of OpenVINO +will not work with these instructions, make sure to +:doc:`install OpenVINO GenAI <../../get-started/install-openvino/install-openvino-genai>`. + +.. note:: + + The examples use the CPU as the target device, however, the GPU is also supported. + Note that for the LLM pipeline, the GPU is used only for inference, while token selection, tokenization, and + detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run + on the CPU. + +1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example: + + .. code-block:: python + + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" + + *Optional*. Optimize the model: + + The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, + it is recommended to use lower precision for model weights, such as INT4, and to compress weights + using NNCF during model export directly: + + .. code-block:: python + + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + +2. Perform generation using the new GenAI API: + + .. tab-set:: + + .. tab-item:: Python + :sync: py + + .. code-block:: python + + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path, "CPU") + print(pipe.generate("The Sun is yellow because")) + + .. tab-item:: C++ + :sync: cpp + + .. code-block:: cpp + + #include "openvino/genai/llm_pipeline.hpp" + #include + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU + std::cout << pipe.generate("The Sun is yellow because"); //input context + +The `LLMPipeline` is the main object used for decoding. You can construct it directly from the +folder with the converted model. It will automatically load the main model, tokenizer, detokenizer, +and the default generation configuration. + +Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information +necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that +its results match those generated by Hugging Face. + +Streaming the Output +########################### + +For more interactive UIs during generation, streaming of model output tokens is supported. See the example +below, where a lambda function outputs words to the console immediately upon generation: + +.. tab-set:: + + .. tab-item:: C++ + + .. code-block:: cpp + + #include "openvino/genai/llm_pipeline.hpp" + #include + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU"); + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + std::cout << pipe.generate("The Sun is yellow because", streamer); + } + +You can also create your custom streamer for more sophisticated processing: + +.. tab-set:: + + .. tab-item:: C++ + + .. code-block:: cpp + + #include + + class CustomStreamer: publict StreamerBase { + public: + void put(int64_t token) {/* decode tokens and do process them*/}; + + void end() {/* decode tokens and do process them*/}; + }; + + int main(int argc, char* argv[]) { + CustomStreamer custom_streamer; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + cout << pipe.generate("The Sun is yellow because", custom_streamer); + } + +Optimizing the Chat Scenario +################################ + +For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs +may prove beneficial. The chat-specific methods **start_chat** and **finish_chat** are used to +mark a conversation session, as you can see in these simple examples: + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. code-block:: python + + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path) + + config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1} + pipe.set_generation_cofnig(config) + + pipe.start_chat() + while True: +     print('question:') +     prompt = input() + if prompt == 'Stop!': +         break +     print(pipe(prompt)) + pipe.finish_chat() + + + .. tab-item:: C++ + :sync: cpp + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string prompt; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + std::cout << "question:\n"; + std::getline(std::cin, prompt); + + std::cout << pipe(prompt) << std::endl>>; + } + pipe.finish_chat(); + } + +Optimizing Generation with Grouped Beam Search +####################################################### + +Leverage grouped beam search decoding and configure generation_config for better text generation +quality and efficient batch processing in GenAI applications. + +Use grouped beam search decoding: + +.. tab-set:: + + .. tab-item:: C++ + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + cout << pipe.generate("The Sun is yellow because", config); + } + +Specify generation_config to use grouped beam search: + +.. tab-set:: + + .. tab-item:: C++ + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string prompt; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + + std::cout << "question:\n"; + cout << prompt << endl; + + auto answer = pipe(prompt, config, streamer); + // no need to print answer, streamer will do that + } + pipe.finish_chat(); + } + +Comparing with Hugging Face Results +####################################### + +Compare and analyze results with those generated by Hugging Face models. + +.. tab-set:: + + .. tab-item:: Python + + .. code-block:: python + + from transformers import AutoTokenizer, AutoModelForCausalLM + + tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + + max_new_tokens = 32 + prompt = 'table is made of' + + encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) + hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) + hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) + print(f'hf_output: {hf_output}') + + import sys + sys.path.append('build-Debug/') + import py_generate_pipeline as genai # set more friendly module name + + pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') + ov_output = pipe(prompt, max_new_tokens=max_new_tokens) + print(f'ov_output: {ov_output}') + + assert hf_output == ov_output + +GenAI API +####################################### + +OpenVINO GenAI Flavor includes the following API: + +* generation_config - defines a configuration class for text generation, enabling customization of the generation process such as the maximum length of the generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding strategy (greedy, beam search, or multinomial sampling). + +* llm_pipeline - provides classes and utilities for text generation, including a pipeline for processing inputs, generating text, and managing outputs with configurable options. + +* streamer_base - an abstract base class for creating streamers. + +* tokenizer - the tokenizer class for text encoding and decoding. + +* visibility - controls the visibility of the GenAI library. + +Learn more about API in the `GenAI repository `__. + +Additional Resources +#################### + +* `OpenVINO GenAI Repo `__ +* `OpenVINO Tokenizers `__ +* `Neural Network Compression Framework `__ + + + diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst index aa5ad1066b5..a0edebf83d1 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst @@ -1,6 +1,6 @@ .. {#llm_inference} -Inference with Hugging Face and Optimum Intel +Run LLMs with Hugging Face and Optimum Intel ===================================================== The steps below show how to load and infer LLMs from Hugging Face using Optimum Intel. diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst index 33b1310fe65..12850aa5ab3 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst @@ -1,6 +1,6 @@ .. {#llm_inference_native_ov} -Inference with Native OpenVINO +Run LLMs with Base OpenVINO =============================== To run Generative AI models using native OpenVINO APIs you need to follow regular **Convert -> Optimize -> Deploy** path with a few simplifications. @@ -21,7 +21,6 @@ Inference code that uses native API cannot benefit from Hugging Face pipelines. To write such pipelines, you can follow the examples provided as part of OpenVINO: -* `Text generation C++ samples that support most popular models like LLaMA 2 `__ * `OpenVINO Latent Consistency Model C++ image generation pipeline `__ * `OpenVINO Stable Diffusion (with LoRA) C++ image generation pipeline `__ @@ -174,7 +173,6 @@ This step is essential for interpreting the model's output. Additional Resources #################### -* `Text generation C++ samples that support most popular models like LLaMA 2 `__ * `OpenVINO GenAI Repo `__ * `OpenVINO Tokenizers `__ * `Neural Network Compression Framework `__