Merge 4ca9fa58b6 into 23fc29a6db

docs/articles_en/get-started/install-openvino.rst

@@ -14,7 +14,7 @@ Install OpenVINO™ 2024.2
    OpenVINO Runtime on Linux <install-openvino/install-openvino-linux>
    OpenVINO Runtime on Windows <install-openvino/install-openvino-windows>
    OpenVINO Runtime on macOS <install-openvino/install-openvino-macos>
-
+   OpenVINO GenAI Flavor <install-openvino/install-openvino-genai>
 
 .. raw:: html
 
@@ -22,17 +22,7 @@ Install OpenVINO™ 2024.2
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <iframe id="selector" src="../_static/selector-tool/selector-1c16038.html" style="width: 100%; border: none" title="Download Intel® Distribution of OpenVINO™ Toolkit"></iframe>
 
-.. warning::
+.. dropdown:: Supported OpenVINO Versions
-
-   The OpenVINO™ Development Tools package has been deprecated and removed from the default
-   installation options. For new projects, the OpenVINO runtime package now includes
-   all necessary components.
-
-   The OpenVINO Development Tools is still available for older versions of OpenVINO,
-   as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`.
-
-
-.. tip::
 
    OpenVINO 2024.2, described here, is not a Long-Term-Support version!
    All currently supported versions are:
@@ -43,6 +33,8 @@ Install OpenVINO™ 2024.2
 
    Moreover, different OpenVINO distributions may support slightly different sets of features.
    Read installation guides for particular distributions for more details.
+   Refer to the :doc:`OpenVINO Release Policy <../../../about-openvino/release-notes-openvino/release-policy>`
+   to learn more about the release types.
 
    .. dropdown:: Distribution Comparison for OpenVINO 2024.2
 
@@ -57,9 +49,27 @@ Install OpenVINO™ 2024.2
       | \* **Of the Linux systems, versions 22.04 and 24.04 include drivers for NPU.**
       |  **For Windows, CPU inference on ARM64 is not supported.**
 
-| **Build OpenVINO from source**
+.. dropdown:: Effortless GenAI integration with OpenVINO GenAI Flavor
-|    OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
+
-     follow the `OpenVINO Build Instructions <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>`__.
+   A new OpenVINO GenAI Flavor streamlines application development by providing
+   LLM-specific interfaces for easy integration of language models, handling tokenization and
+   text generation. For installation and usage instructions, proceed to
+   :doc:`Install OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>` and
+   :doc:`Run LLMs with OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>`.
+
+.. dropdown:: Deprecation of OpenVINO™ Development Tools Package
+
+   The OpenVINO™ Development Tools package has been deprecated and removed from the default
+   installation options. For new projects, the OpenVINO runtime package now includes
+   all necessary components.
+
+   The OpenVINO Development Tools is still available for older versions of OpenVINO,
+   as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`.
+
+.. dropdown:: Building OpenVINO from source
+
+   OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
+   follow the `OpenVINO Build Instructions <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>`__.
 
 
 

docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst

@@ -32,7 +32,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File
        RHEL8 x86_64          V      V     n/a
       ===================  =====  =====  =====
 
-
 .. tab-set::
 
    .. tab-item:: System Requirements
@@ -94,8 +93,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File
 
 
 
-
-
 Installing OpenVINO Runtime
 ############################################################
 

docs/articles_en/get-started/install-openvino/install-openvino-genai.rst

@@ -0,0 +1,62 @@
+Install OpenVINO™ GenAI
+====================================
+
+OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of generative AI models.
+It hides the complexity of the generation process and minimizes the amount of code required.
+You can now provide a model and input context directly to OpenVINO, which performs tokenization of the
+input text, executes the generation loop on the selected device, and returns the generated text.
+For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`.
+
+To see GenAI in action, check the Jupyter notebooks:
+`LLM-powered Chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/README.md>`__ and
+`LLM Instruction-following pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-question-answering/README.md>`__
+
+The OpenVINO GenAI flavor is available for installation via Archive and PyPI distributions:
+
+Archive Installation
+###############################
+
+To install the GenAI flavor of OpenVINO from an archive file, follow the standard installation steps for your system
+but instead of using the vanilla package file, download the one with OpenVINO GenAI:
+
+.. tab-set::
+
+   .. tab-item:: x86_64
+      :sync: x86-64
+
+      .. tab-set::
+
+         .. tab-item:: Ubuntu 24.04
+            :sync: ubuntu-24
+
+            .. code-block:: sh
+
+
+               curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz
+               tar -xf openvino_2024.1.0.tgz
+               sudo mv l_openvino_toolkit_ubuntu24_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0
+
+         .. tab-item:: Ubuntu 22.04
+            :sync: ubuntu-22
+
+            .. code-block:: sh
+
+               curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz
+               tar -xf openvino_2024.1.0.tgz
+               sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0
+
+Here are the full guides:
+:doc:`Linux <install-openvino-archive-linux>`,
+:doc:`Windows <install-openvino-archive-windows>`, and
+:doc:`macOS <install-openvino-archive-macos>`.
+
+PyPI Installation
+###############################
+
+To install the GenAI flavor of OpenVINO via PyPI, follow the standard :doc:`installation steps <install-openvino-pip>`,
+but use the *openvino-genai* package instead of *openvino*:
+
+.. code-block:: python
+
+   python -m pip install openvino-genai
+

docs/articles_en/get-started/install-openvino/install-openvino-linux.rst

@@ -26,7 +26,6 @@ Install OpenVINO™ Runtime on Linux
    Use Docker <install-openvino-docker-linux>
    Use npm <install-openvino-npm>
 
-
 If you want to install OpenVINO™ Runtime on Linux, you have the following options:
 
 * :doc:`Install OpenVINO using an Archive File <install-openvino-archive-linux>`

docs/articles_en/get-started/install-openvino/install-openvino-pip.rst

@@ -100,13 +100,12 @@ Use the following command:
 Step 4. Install the Package
 +++++++++++++++++++++++++++
 
-Use the following command:
+Use the following command to install OpenVINO Base or OpenVINO GenAI package:
 
-.. code-block:: sh
+.. code-block:: python
 
    python -m pip install openvino
 
-
 Step 5. Verify that the Package Is Installed
 ++++++++++++++++++++++++++++++++++++++++++++
 

docs/articles_en/learn-openvino/llm_inference_guide.rst

@@ -13,8 +13,9 @@ Large Language Model Inference Guide
    :maxdepth: 1
    :hidden:
 
-   LLM Inference with Optimum Intel <llm_inference_guide/llm-inference-hf>
+   Run LLMs with Optimum Intel <llm_inference_guide/llm-inference-hf>
-   LLM Inference with OpenVINO API <llm_inference_guide/llm-inference-native-ov>
+   Run LLMs with OpenVINO GenAI Flavor <llm_inference_guide/genai-guide>
+   Run LLMs with Base OpenVINO <llm_inference_guide/llm-inference-native-ov>
    OpenVINO Tokenizers <llm_inference_guide/ov-tokenizers>
 
 Large Language Models (LLMs) like GPT are transformative deep learning networks capable of a
@@ -39,12 +40,13 @@ The advantages of using OpenVINO for LLM deployment:
 
 *	**Provides stateful model optimization**: models from the Hugging Face Transformers are converted into a stateful form, optimizing inference performance and memory usage in long-running text generation tasks by managing past KV-cache tensors more efficiently internally. This feature is automatically activated for many supported models, while unsupported ones remain stateless. Learn more about the :doc:`Stateful models and State API <../openvino-workflow/running-inference/stateful-models>`.
 
-OpenVINO offers two main paths for Generative AI use cases:
+OpenVINO offers three main paths for Generative AI use cases:
 
+* **OpenVINO GenAI Flavor**: use OpenVINO GenAI APIs (Python and C++).
 * **Hugging Face**: use OpenVINO as a backend for Hugging Face frameworks (transformers,
   diffusers) through the `Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__
   extension.
-* **Native OpenVINO**: use OpenVINO native APIs (Python and C++) with
+* **Base OpenVINO**: use OpenVINO native APIs (Python and C++) with
   `custom pipeline code <https://github.com/openvinotoolkit/openvino.genai>`__.
 
 In both cases, the OpenVINO runtime is used for inference, and OpenVINO tools are used for

docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst

@@ -0,0 +1,287 @@
+Run LLMs with OpenVINO GenAI Flavor
+=====================================
+
+.. meta::
+   :description: Learn how to use the OpenVINO GenAI flavor to execute LLM models.
+
+This guide will show you how to integrate the OpenVINO GenAI flavor into your application, covering
+loading a model and passing the input context to receive generated text. Note that the vanilla flavor of OpenVINO
+will not work with these instructions, make sure to
+:doc:`install OpenVINO GenAI <../../get-started/install-openvino/install-openvino-genai>`.
+
+.. note::
+
+   The examples use the CPU as the target device, however, the GPU is also supported.
+   Note that for the LLM pipeline, the GPU is used only for inference, while token selection, tokenization, and
+   detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run
+   on the CPU.
+
+1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example:
+
+   .. code-block:: python
+
+      optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"
+
+   *Optional*. Optimize the model:
+
+   The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance,
+   it is recommended to use lower precision for model weights, such as INT4, and to compress weights
+   using NNCF during model export directly:
+
+   .. code-block:: python
+
+      optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code
+
+2. Perform generation using the new GenAI API:
+
+   .. tab-set::
+
+      .. tab-item:: Python
+         :sync: py
+
+         .. code-block:: python
+
+            import openvino_genai as ov_genai
+            pipe = ov_genai.LLMPipeline(model_path, "CPU")
+            print(pipe.generate("The Sun is yellow because"))
+
+      .. tab-item:: C++
+         :sync: cpp
+
+         .. code-block:: cpp
+
+            #include "openvino/genai/llm_pipeline.hpp"
+            #include <iostream>
+
+            int main(int argc, char* argv[]) {
+            std::string model_path = argv[1];
+            ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU
+            std::cout << pipe.generate("The Sun is yellow because"); //input context
+
+The `LLMPipeline` is the main object used for decoding. You can construct it directly from the
+folder with the converted model. It will automatically load the main model, tokenizer, detokenizer,
+and the default generation configuration.
+
+Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information
+necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that
+its results match those generated by Hugging Face.
+
+Streaming the Output
+###########################
+
+For more interactive UIs during generation, streaming of model output tokens is supported. See the example
+below, where a lambda function outputs words to the console immediately upon generation:
+
+.. tab-set::
+
+   .. tab-item:: C++
+
+      .. code-block:: cpp
+
+         #include "openvino/genai/llm_pipeline.hpp"
+         #include <iostream>
+
+         int main(int argc, char* argv[]) {
+            std::string model_path = argv[1];
+            ov::genai::LLMPipeline pipe(model_path, "CPU");
+
+            auto streamer = [](std::string word) { std::cout << word << std::flush; };
+            std::cout << pipe.generate("The Sun is yellow because", streamer);
+         }
+
+You can also create your custom streamer for more sophisticated processing:
+
+.. tab-set::
+
+   .. tab-item:: C++
+
+      .. code-block:: cpp
+
+         #include <streamer_base.hpp>
+
+         class CustomStreamer: publict StreamerBase {
+         public:
+            void put(int64_t token) {/* decode tokens and do process them*/};
+
+            void end() {/* decode tokens and do process them*/};
+         };
+
+         int main(int argc, char* argv[]) {
+            CustomStreamer custom_streamer;
+
+            std::string model_path = argv[1];
+            ov::LLMPipeline pipe(model_path, "CPU");
+            cout << pipe.generate("The Sun is yellow because", custom_streamer);
+         }
+
+Optimizing the Chat Scenario
+################################
+
+For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs
+may prove beneficial. The chat-specific methods **start_chat** and **finish_chat** are used to
+mark a conversation session, as you can see in these simple examples:
+
+.. tab-set::
+
+   .. tab-item:: Python
+      :sync: py
+
+      .. code-block:: python
+
+         import openvino_genai as ov_genai
+         pipe = ov_genai.LLMPipeline(model_path)
+
+         config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1}
+         pipe.set_generation_cofnig(config)
+
+         pipe.start_chat()
+         while True:
+             print('question:')
+             prompt = input()
+            if prompt == 'Stop!':
+                 break
+             print(pipe(prompt))
+         pipe.finish_chat()
+
+
+   .. tab-item:: C++
+      :sync: cpp
+
+      .. code-block:: cpp
+
+         int main(int argc, char* argv[]) {
+            std::string prompt;
+
+            std::string model_path = argv[1];
+            ov::LLMPipeline pipe(model_path, "CPU");
+
+            pipe.start_chat();
+            for (size_t i = 0; i < questions.size(); i++) {
+               std::cout << "question:\n";
+               std::getline(std::cin, prompt);
+
+               std::cout << pipe(prompt) << std::endl>>;
+            }
+            pipe.finish_chat();
+         }
+
+Optimizing Generation with Grouped Beam Search
+#######################################################
+
+Leverage grouped beam search decoding and configure generation_config for better text generation
+quality and efficient batch processing in GenAI applications.
+
+Use grouped beam search decoding:
+
+.. tab-set::
+
+   .. tab-item:: C++
+
+      .. code-block:: cpp
+
+         int main(int argc, char* argv[]) {
+            std::string model_path = argv[1];
+            ov::LLMPipeline pipe(model_path, "CPU");
+            ov::GenerationConfig config = pipe.get_generation_config();
+            config.max_new_tokens = 256;
+            config.num_groups = 3;
+            config.group_size = 5;
+            config.diversity_penalty = 1.0f;
+
+            cout << pipe.generate("The Sun is yellow because", config);
+         }
+
+Specify generation_config to use grouped beam search:
+
+.. tab-set::
+
+   .. tab-item:: C++
+
+      .. code-block:: cpp
+
+         int main(int argc, char* argv[]) {
+            std::string prompt;
+
+            std::string model_path = argv[1];
+            ov::LLMPipeline pipe(model_path, "CPU");
+
+            ov::GenerationConfig config = pipe.get_generation_config();
+            config.max_new_tokens = 256;
+            config.num_groups = 3;
+            config.group_size = 5;
+            config.diversity_penalty = 1.0f;
+
+            auto streamer = [](std::string word) { std::cout << word << std::flush; };
+
+            pipe.start_chat();
+            for (size_t i = 0; i < questions.size(); i++) {
+
+               std::cout << "question:\n";
+               cout << prompt << endl;
+
+               auto answer = pipe(prompt, config, streamer);
+               // no need to print answer, streamer will do that
+            }
+            pipe.finish_chat();
+         }
+
+Comparing with Hugging Face Results
+#######################################
+
+Compare and analyze results with those generated by Hugging Face models.
+
+.. tab-set::
+
+   .. tab-item:: Python
+
+      .. code-block:: python
+
+         from transformers import AutoTokenizer, AutoModelForCausalLM
+
+         tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
+         model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
+
+         max_new_tokens = 32
+         prompt = 'table is made of'
+
+         encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False)
+         hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False)
+         hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:])
+         print(f'hf_output: {hf_output}')
+
+         import sys
+         sys.path.append('build-Debug/')
+         import py_generate_pipeline as genai # set more friendly module name
+
+         pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/')
+         ov_output = pipe(prompt, max_new_tokens=max_new_tokens)
+         print(f'ov_output: {ov_output}')
+
+         assert hf_output == ov_output
+
+GenAI API
+#######################################
+
+OpenVINO GenAI Flavor includes the following API:
+
+* generation_config - defines a configuration class for text generation, enabling customization of the generation process such as the maximum length of the generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding strategy (greedy, beam search, or multinomial sampling).
+
+* llm_pipeline - provides classes and utilities for text generation, including a pipeline for processing inputs, generating text, and managing outputs with configurable options.
+
+* streamer_base - an abstract base class for creating streamers.
+
+* tokenizer - the tokenizer class for text encoding and decoding.
+
+* visibility  -  controls the visibility of the GenAI library.
+
+Learn more about API in the `GenAI repository <https://github.com/openvinotoolkit/openvino.genai/tree/master/src/cpp/include/openvino/genai>`__.
+
+Additional Resources
+####################
+
+* `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__
+* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
+* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__
+
+
+

docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst

@@ -1,6 +1,6 @@
 .. {#llm_inference}
 
-Inference with Hugging Face and Optimum Intel
+Run LLMs with Hugging Face and Optimum Intel
 =====================================================
 
 The steps below show how to load and infer LLMs from Hugging Face using Optimum Intel.

docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst

@@ -1,6 +1,6 @@
 .. {#llm_inference_native_ov}
 
-Inference with Native OpenVINO
+Run LLMs with Base OpenVINO
 ===============================
 
 To run Generative AI models using native OpenVINO APIs you need to follow regular **Convert -> Optimize -> Deploy** path with a few simplifications.
@@ -21,7 +21,6 @@ Inference code that uses native API cannot benefit from Hugging Face pipelines.
 
 To write such pipelines, you can follow the examples provided as part of OpenVINO:
 
-* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
 * `OpenVINO Latent Consistency Model C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/lcm_dreamshaper_v7/cpp>`__
 * `OpenVINO Stable Diffusion (with LoRA) C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/stable_diffusion_1_5/cpp>`__
 
@@ -174,7 +173,6 @@ This step is essential for interpreting the model's output.
 Additional Resources
 ####################
 
-* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
 * `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__
 * `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__
 * `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__