From 0ba9c1a723fe68d3081cf8899bd8d2f22086c65a Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Tue, 4 Jun 2024 14:38:17 +0200 Subject: [PATCH 01/21] add initial pages --- .../[legacy]-supported-model-formats.rst | 2 +- .../get-started/install-openvino.rst | 1 + .../install-openvino-genai.rst | 27 +++++++ .../learn-openvino/llm_inference_guide.rst | 1 + .../llm_inference_guide/genai-guide.rst | 71 +++++++++++++++++++ 5 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 docs/articles_en/get-started/install-openvino/install-openvino-genai.rst create mode 100644 docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst diff --git a/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst b/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst index b4296d69a8a..9cff382660c 100644 --- a/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst +++ b/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst @@ -48,7 +48,7 @@ Here are code examples of how to use these methods with different model formats: .. tab-set:: .. tab-item:: Python - :sync: py + py * The ``convert_model()`` method: diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index 70c53b53891..6f67247d0b8 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -14,6 +14,7 @@ Install OpenVINO™ 2024.2 OpenVINO Runtime on Linux OpenVINO Runtime on Windows OpenVINO Runtime on macOS + OpenVINO GenAI .. raw:: html diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst new file mode 100644 index 00000000000..13c9669352c --- /dev/null +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -0,0 +1,27 @@ + +Install OpenVINO™ GenAI Package +==================================== + +.. meta:: + :description: + + +The new GenAI API provides a set of LLM-specific interfaces designed to facilitate the integration +of language models into applications. This API hides the complexity of the generation process, +such as tokenization and managing generation loops, and significantly minimizes the amount of code needed for the application to work. +Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs +tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. +The GenAI API is available through the following distribution channels: + +1. PyPI + +.. code-block:: sh + + pip install openvino-genai + +2. Archives + + +For a quickstart guide on how to use the OpenVINO GenAI API, read the :doc:`GenAI API Guide `. +Refer to the GenAI API reference page for more detailed information. + diff --git a/docs/articles_en/learn-openvino/llm_inference_guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide.rst index b26c3adc74d..e7931209b9d 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide.rst @@ -13,6 +13,7 @@ Large Language Model Inference Guide :maxdepth: 1 :hidden: + OpenVINO GenAI Guide LLM Inference with Optimum Intel LLM Inference with OpenVINO API OpenVINO Tokenizers diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst new file mode 100644 index 00000000000..c7476000fbf --- /dev/null +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -0,0 +1,71 @@ + +OpenVINO GenAI Guide +=============================== + +This guide will walk through the essential steps for integrating the OpenVINO GenAI API into your application. +The steps below show the initial setup, demonstrate how to load a model, +and illustrate the process of passing the input context to receive generated text. + +1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama is used for this example: + +.. code-block:: python + + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code + +Optional. Optimize the model: + +This model will be in the form of an optimized OpenVINO IR of the fp16 precision. +To make LLM inference more performant we recommend using a lower precision for model weights, +i.e. int4, and compress weights using NNCF during model export directly: + +.. code-block:: python + + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + +2. Perform generation using the new GenAI API: + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. code-block:: python + + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path, "CPU") + print(pipe.generate("The Sun is yellow bacause")) + + .. tab-item:: C++ + :sync: cpp + + .. code-block:: cpp + + #include "openvino/genai/llm_pipeline.hpp" + #include + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU + std::cout << pipe.generate("The Sun is yellow bacause"); //input context + ++OUTPUT + +Once the model is exported from Hugging Face Optimum-Intel, it already contains all the necessary +information for execution, including the tokenizer/detokenizer and the generation config +ensuring that its results match Hugging Face generation. + + + + + +Additional Resources +#################### + +* `Text generation C++ samples that support most popular models like LLaMA 2 `__ +* `OpenVINO GenAI Repo `__ +* `OpenVINO Tokenizers `__ +* `Neural Network Compression Framework `__ +* :doc:`Stateful Models Low-Level Details <../../openvino-workflow/running-inference/stateful-models>` +* :doc:`Working with Textual Data <../../openvino-workflow/running-inference/string-tensors>` + + From a15d3e8357cfafa70d1af9c5cf32d7e6ba54c80c Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Tue, 4 Jun 2024 14:42:49 +0200 Subject: [PATCH 02/21] remove accidental change --- .../legacy-conversion-api/[legacy]-supported-model-formats.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst b/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst index 9cff382660c..b4296d69a8a 100644 --- a/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst +++ b/docs/articles_en/documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats.rst @@ -48,7 +48,7 @@ Here are code examples of how to use these methods with different model formats: .. tab-set:: .. tab-item:: Python - py + :sync: py * The ``convert_model()`` method: From 94e7418bdec1e28c30a2e35646cce703333854f1 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Wed, 5 Jun 2024 16:53:16 +0200 Subject: [PATCH 03/21] add more examples --- .../get-started/install-openvino.rst | 8 +- .../install-openvino-archive-linux.rst | 5 +- .../install-openvino-genai.rst | 27 -- .../install-openvino-linux.rst | 3 + .../install-openvino/install-openvino-pip.rst | 21 +- .../llm_inference_guide/genai-guide.rst | 232 +++++++++++++++--- 6 files changed, 233 insertions(+), 63 deletions(-) delete mode 100644 docs/articles_en/get-started/install-openvino/install-openvino-genai.rst diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index 6f67247d0b8..ed3e5a71c2f 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -14,7 +14,6 @@ Install OpenVINO™ 2024.2 OpenVINO Runtime on Linux OpenVINO Runtime on Windows OpenVINO Runtime on macOS - OpenVINO GenAI .. raw:: html @@ -23,6 +22,13 @@ Install OpenVINO™ 2024.2 +The new OpenVINO GenAI API package provides a set of LLM-specific interfaces to facilitate the integration +of language models into applications. This API hides the complexity of the generation process +and significantly minimizes the amount of code needed for the application to work. +Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs +tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. +For a quickstart guide on how to use the OpenVINO GenAI API, refer to the :doc:`GenAI API Guide <../learn-openvino/llm_inference_guide/genai-guide>` or the GenAI API reference page. + .. warning:: The OpenVINO™ Development Tools package has been deprecated and removed from the default diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst index 32b7b36ce5b..30b0cd5ac88 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst @@ -32,6 +32,9 @@ Install OpenVINO™ Runtime on Linux from an Archive File RHEL8 x86_64 V V n/a =================== ===== ===== ===== +.. tip:: + + The new OpenVINO GenAI package is now available for installation via :doc:`Archives ` distribution. .. tab-set:: @@ -94,8 +97,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File - - Installing OpenVINO Runtime ############################################################ diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst deleted file mode 100644 index 13c9669352c..00000000000 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ /dev/null @@ -1,27 +0,0 @@ - -Install OpenVINO™ GenAI Package -==================================== - -.. meta:: - :description: - - -The new GenAI API provides a set of LLM-specific interfaces designed to facilitate the integration -of language models into applications. This API hides the complexity of the generation process, -such as tokenization and managing generation loops, and significantly minimizes the amount of code needed for the application to work. -Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs -tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. -The GenAI API is available through the following distribution channels: - -1. PyPI - -.. code-block:: sh - - pip install openvino-genai - -2. Archives - - -For a quickstart guide on how to use the OpenVINO GenAI API, read the :doc:`GenAI API Guide `. -Refer to the GenAI API reference page for more detailed information. - diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst index 38561aaebae..94b4f13e22f 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst @@ -26,6 +26,9 @@ Install OpenVINO™ Runtime on Linux Use Docker Use npm +.. tip:: + + The new OpenVINO GenAI package is now available for installation via :doc:`Archives ` and :doc:`PyPI ` distributions. If you want to install OpenVINO™ Runtime on Linux, you have the following options: diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst index ed3c1b12428..186c7948820 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst @@ -18,6 +18,10 @@ Install Intel® Distribution of OpenVINO™ Toolkit from PyPI Repository (all x86_64 / arm64 architectures) * macOS offers support only for CPU inference +.. tip:: + + The new OpenVINO GenAI package is now available for installation via :doc:`PyPI ` distribution. + .. tab-set:: .. tab-item:: System Requirements @@ -100,11 +104,22 @@ Use the following command: Step 4. Install the Package +++++++++++++++++++++++++++ -Use the following command: +Use the following command to install OpenVINO Base or OpenVINO GenAI package: -.. code-block:: sh - python -m pip install openvino +.. tab-set:: + + .. tab-item:: OpenVINO Base Package + + .. code-block:: python + + python -m pip install openvino + + .. tab-item:: OpenVINO with GenAI + + .. code-block:: python + + python -m pip install openvino-genai Step 5. Verify that the Package Is Installed diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index c7476000fbf..f6d8651caea 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -1,71 +1,243 @@ -OpenVINO GenAI Guide +OpenVINO GenAI API Guide =============================== This guide will walk through the essential steps for integrating the OpenVINO GenAI API into your application. -The steps below show the initial setup, demonstrate how to load a model, +The steps below demonstrate how to load a model, and illustrate the process of passing the input context to receive generated text. +The examples below use a CPU as the target device, however, the GPU support is also available. +Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. +Tokenizers are represented as a separate model and run on the CPU using the provided inference capabilities. + +Install OpenVINO GenAI API using :doc:`PyPI or Archives <../../../install-openvino>`. + 1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama is used for this example: -.. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" Optional. Optimize the model: -This model will be in the form of an optimized OpenVINO IR of the fp16 precision. -To make LLM inference more performant we recommend using a lower precision for model weights, -i.e. int4, and compress weights using NNCF during model export directly: +The model is an optimized OpenVINO IR with fp16 precision. For enhanced LLM performance, +it is recommended to use lower precision for model weights, such as int4, and to compress weights +using NNCF during model export directly: -.. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code 2. Perform generation using the new GenAI API: -.. tab-set:: + .. tab-set:: - .. tab-item:: Python - :sync: py + .. tab-item:: Python + :sync: py - .. code-block:: python + .. code-block:: python - import openvino_genai as ov_genai - pipe = ov_genai.LLMPipeline(model_path, "CPU") - print(pipe.generate("The Sun is yellow bacause")) + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path, "CPU") + print(pipe.generate("The Sun is yellow because")) - .. tab-item:: C++ - :sync: cpp + .. tab-item:: C++ + :sync: cpp - .. code-block:: cpp + .. code-block:: cpp - #include "openvino/genai/llm_pipeline.hpp" - #include + #include "openvino/genai/llm_pipeline.hpp" + #include - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU - std::cout << pipe.generate("The Sun is yellow bacause"); //input context - -+OUTPUT + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU + std::cout << pipe.generate("The Sun is yellow because"); //input context Once the model is exported from Hugging Face Optimum-Intel, it already contains all the necessary information for execution, including the tokenizer/detokenizer and the generation config ensuring that its results match Hugging Face generation. +Streaming Options +########################### +For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: + +.. code-block:: cpp + + #include "openvino/genai/llm_pipeline.hpp" + #include + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU"); + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + std::cout << pipe.generate("The Sun is yellow bacause", streamer); + } + +You can also create your custom streamer for more sophisticated processing: + +.. code-block:: cpp + + #include + + class CustomStreamer: publict StreamerBase { + public: + void put(int64_t token) {/* decode tokens and do process them*/}; + + void end() {/* decode tokens and do process them*/}; + }; + + int main(int argc, char* argv[]) { + CustomStreamer custom_streamer; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + cout << pipe.generate("The Sun is yellow bacause", custom_streamer); + } + +Chat Scenarios Optimization +############################## + +For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs +offers optimization benefits. The chat-specific methods **start_chat** and **finish_chat** are used to +mark a conversation session. Simplified Python and C++ examples are provided below: + + .. tab-set:: + + .. tab-item:: Python + :sync: py + + .. code-block:: python + + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path) + + config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1} + pipe.set_generation_cofnig(config) + + pipe.start_chat() + while True: +     print('question:') +     prompt = input() + if prompt == 'Stop!': +         break +     print(pipe(prompt)) + pipe.finish_chat() + + + .. tab-item:: C++ + :sync: cpp + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string prompt; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + std::cout << "question:\n"; + std::getline(std::cin, prompt); + + std::cout << pipe(prompt) << std::endl>>; + } + pipe.finish_chat(); +} + +Additional Settings +############################## + +C++ Features +++++++++++++++ + +.. dropdown:: Using Group Beam Search Decoding + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + cout << pipe.generate("The Sun is yellow bacause", config); + } + +.. dropdown:: Specifying generation_config to Use Grouped Beam Search + + .. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string prompt; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + + std::cout << "question:\n"; + cout << prompt << endl; + + auto answer = pipe(prompt, config, streamer); + // no need to print answer, streamer will do that + } + pipe.finish_chat(); + } + +Python Features +++++++++++++++++++ + +.. dropdown:: Test to Compare the Results with Hugging Face Outputs + + .. code-block:: python + + from transformers import AutoTokenizer, AutoModelForCausalLM + + tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + + max_new_tokens = 32 + prompt = 'table is made of' + + encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) + hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) + hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) + print(f'hf_output: {hf_output}') + + import sys + sys.path.append('build-Debug/') + import py_generate_pipeline as genai # set more friendly module name + + pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') + ov_output = pipe(prompt, max_new_tokens=max_new_tokens) + print(f'ov_output: {ov_output}') + + assert hf_output == ov_output Additional Resources #################### -* `Text generation C++ samples that support most popular models like LLaMA 2 `__ * `OpenVINO GenAI Repo `__ * `OpenVINO Tokenizers `__ * `Neural Network Compression Framework `__ -* :doc:`Stateful Models Low-Level Details <../../openvino-workflow/running-inference/stateful-models>` -* :doc:`Working with Textual Data <../../openvino-workflow/running-inference/string-tensors>` + From 1ed010075dd604db00f147401ba5dd204a3385cf Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Wed, 5 Jun 2024 21:56:32 +0200 Subject: [PATCH 04/21] restructuring --- .../get-started/install-openvino.rst | 14 +- .../install-openvino-archive-linux.rst | 2 +- .../install-openvino/install-openvino-pip.rst | 2 +- .../llm_inference_guide/genai-guide.rst | 246 +++++++++--------- 4 files changed, 132 insertions(+), 132 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index ed3e5a71c2f..14e1443df59 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -22,12 +22,14 @@ Install OpenVINO™ 2024.2 -The new OpenVINO GenAI API package provides a set of LLM-specific interfaces to facilitate the integration -of language models into applications. This API hides the complexity of the generation process -and significantly minimizes the amount of code needed for the application to work. -Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs -tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. -For a quickstart guide on how to use the OpenVINO GenAI API, refer to the :doc:`GenAI API Guide <../learn-openvino/llm_inference_guide/genai-guide>` or the GenAI API reference page. +.. tip:: + + The new OpenVINO GenAI API package provides a set of LLM-specific interfaces to facilitate the integration + of language models into applications. This API hides the complexity of the generation process + and significantly minimizes the amount of code needed for the application to work. + Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs + tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. + For a quickstart guide, refer to the :doc:`GenAI API Guide <../learn-openvino/llm_inference_guide/genai-guide>` or the GenAI API reference page. .. warning:: diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst index 30b0cd5ac88..3a4198da56f 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst @@ -34,7 +34,7 @@ Install OpenVINO™ Runtime on Linux from an Archive File .. tip:: - The new OpenVINO GenAI package is now available for installation via :doc:`Archives ` distribution. + The new OpenVINO GenAI package is now available for installation via Archives. Learn more in the Installing OpenVINO Runtime section. .. tab-set:: diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst index 186c7948820..c2301056ca5 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst @@ -20,7 +20,7 @@ Install Intel® Distribution of OpenVINO™ Toolkit from PyPI Repository .. tip:: - The new OpenVINO GenAI package is now available for installation via :doc:`PyPI ` distribution. + The new OpenVINO GenAI package is now available for installation via PyPI. Learn more in the Installing OpenVINO Runtime section. .. tab-set:: diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index f6d8651caea..9d0cf35084f 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -2,57 +2,56 @@ OpenVINO GenAI API Guide =============================== -This guide will walk through the essential steps for integrating the OpenVINO GenAI API into your application. -The steps below demonstrate how to load a model, -and illustrate the process of passing the input context to receive generated text. +This guide shows the essential steps for integrating the OpenVINO GenAI API into your application. +The steps below demonstrate how to load a model and pass the input context to receive generated text. -The examples below use a CPU as the target device, however, the GPU support is also available. +The examples use a CPU as the target device, however, the GPU support is also available. Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. Tokenizers are represented as a separate model and run on the CPU using the provided inference capabilities. -Install OpenVINO GenAI API using :doc:`PyPI or Archives <../../../install-openvino>`. +Before proceeding, make sure that you have installed the OpenVINO GenAI API using :doc:`PyPI or Archives <../../get-started/install-openvino>`. -1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama is used for this example: +1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used for this example: - .. code-block:: python +.. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" -Optional. Optimize the model: +*Optional*. Optimize the model: The model is an optimized OpenVINO IR with fp16 precision. For enhanced LLM performance, it is recommended to use lower precision for model weights, such as int4, and to compress weights using NNCF during model export directly: - .. code-block:: python +.. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code 2. Perform generation using the new GenAI API: - .. tab-set:: +.. tab-set:: - .. tab-item:: Python - :sync: py + .. tab-item:: Python + :sync: py - .. code-block:: python + .. code-block:: python - import openvino_genai as ov_genai - pipe = ov_genai.LLMPipeline(model_path, "CPU") - print(pipe.generate("The Sun is yellow because")) + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path, "CPU") + print(pipe.generate("The Sun is yellow because")) - .. tab-item:: C++ - :sync: cpp + .. tab-item:: C++ + :sync: cpp - .. code-block:: cpp + .. code-block:: cpp - #include "openvino/genai/llm_pipeline.hpp" - #include + #include "openvino/genai/llm_pipeline.hpp" + #include - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU - std::cout << pipe.generate("The Sun is yellow because"); //input context + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU + std::cout << pipe.generate("The Sun is yellow because"); //input context Once the model is exported from Hugging Face Optimum-Intel, it already contains all the necessary information for execution, including the tokenizer/detokenizer and the generation config @@ -73,7 +72,7 @@ For more interactive UIs during generation, streaming of model output tokens is ov::genai::LLMPipeline pipe(model_path, "CPU"); auto streamer = [](std::string word) { std::cout << word << std::flush; }; - std::cout << pipe.generate("The Sun is yellow bacause", streamer); + std::cout << pipe.generate("The Sun is yellow because", streamer); } You can also create your custom streamer for more sophisticated processing: @@ -104,131 +103,130 @@ For chat scenarios where inputs and outputs represent a conversation, maintainin offers optimization benefits. The chat-specific methods **start_chat** and **finish_chat** are used to mark a conversation session. Simplified Python and C++ examples are provided below: - .. tab-set:: +.. tab-set:: - .. tab-item:: Python - :sync: py + .. tab-item:: Python + :sync: py - .. code-block:: python + .. code-block:: python - import openvino_genai as ov_genai - pipe = ov_genai.LLMPipeline(model_path) + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path) - config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1} - pipe.set_generation_cofnig(config) + config = {'num_groups': 3, 'group_size': 5, 'diversity_penalty': 1.1} + pipe.set_generation_cofnig(config) - pipe.start_chat() - while True: -     print('question:') -     prompt = input() - if prompt == 'Stop!': -         break -     print(pipe(prompt)) - pipe.finish_chat() + pipe.start_chat() + while True: +     print('question:') +     prompt = input() + if prompt == 'Stop!': +         break +     print(pipe(prompt)) + pipe.finish_chat() - .. tab-item:: C++ - :sync: cpp + .. tab-item:: C++ + :sync: cpp - .. code-block:: cpp + .. code-block:: cpp - int main(int argc, char* argv[]) { - std::string prompt; + int main(int argc, char* argv[]) { + std::string prompt; - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); - pipe.start_chat(); - for (size_t i = 0; i < questions.size(); i++) { - std::cout << "question:\n"; - std::getline(std::cin, prompt); + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + std::cout << "question:\n"; + std::getline(std::cin, prompt); - std::cout << pipe(prompt) << std::endl>>; - } - pipe.finish_chat(); -} + std::cout << pipe(prompt) << std::endl>>; + } + pipe.finish_chat(); + } -Additional Settings -############################## +Optimizing Text Generation with Group Beam Search +####################################################### -C++ Features -++++++++++++++ +Leverage group beam search decoding and configure generation_config for better text generation quality and efficient batch processing in GenAI applications. -.. dropdown:: Using Group Beam Search Decoding +Use group beam search decoding: - .. code-block:: cpp +.. code-block:: cpp - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; - cout << pipe.generate("The Sun is yellow bacause", config); + cout << pipe.generate("The Sun is yellow bacause", config); + } + +Specify generation_config to use grouped beam search: + +.. code-block:: cpp + + int main(int argc, char* argv[]) { + std::string prompt; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { + + std::cout << "question:\n"; + cout << prompt << endl; + + auto answer = pipe(prompt, config, streamer); + // no need to print answer, streamer will do that } + pipe.finish_chat(); + } -.. dropdown:: Specifying generation_config to Use Grouped Beam Search +Comparing with Hugging Face Results +####################################### - .. code-block:: cpp +Compare and analyze results with those generated by Hugging Face models. - int main(int argc, char* argv[]) { - std::string prompt; +.. code-block:: python - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); + from transformers import AutoTokenizer, AutoModelForCausalLM - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - auto streamer = [](std::string word) { std::cout << word << std::flush; }; + max_new_tokens = 32 + prompt = 'table is made of' - pipe.start_chat(); - for (size_t i = 0; i < questions.size(); i++) { + encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) + hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) + hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) + print(f'hf_output: {hf_output}') - std::cout << "question:\n"; - cout << prompt << endl; + import sys + sys.path.append('build-Debug/') + import py_generate_pipeline as genai # set more friendly module name - auto answer = pipe(prompt, config, streamer); - // no need to print answer, streamer will do that - } - pipe.finish_chat(); - } + pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') + ov_output = pipe(prompt, max_new_tokens=max_new_tokens) + print(f'ov_output: {ov_output}') -Python Features -++++++++++++++++++ - -.. dropdown:: Test to Compare the Results with Hugging Face Outputs - - .. code-block:: python - - from transformers import AutoTokenizer, AutoModelForCausalLM - - tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - - max_new_tokens = 32 - prompt = 'table is made of' - - encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) - hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) - hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) - print(f'hf_output: {hf_output}') - - import sys - sys.path.append('build-Debug/') - import py_generate_pipeline as genai # set more friendly module name - - pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') - ov_output = pipe(prompt, max_new_tokens=max_new_tokens) - print(f'ov_output: {ov_output}') - - assert hf_output == ov_output + assert hf_output == ov_output From 1544132d51a22158391623a591463fb04b6eb6a9 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 6 Jun 2024 11:58:37 +0200 Subject: [PATCH 05/21] add language markers --- .../install-openvino-archive-linux.rst | 2 +- .../install-openvino-linux.rst | 2 +- .../llm_inference_guide/genai-guide.rst | 152 ++++++++++-------- 3 files changed, 83 insertions(+), 73 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst index 3a4198da56f..79fcc312c16 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst @@ -34,7 +34,7 @@ Install OpenVINO™ Runtime on Linux from an Archive File .. tip:: - The new OpenVINO GenAI package is now available for installation via Archives. Learn more in the Installing OpenVINO Runtime section. + The new OpenVINO GenAI package is now available for installation via Archive distribution. Learn more in the Installing OpenVINO Runtime section. .. tab-set:: diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst index 94b4f13e22f..67d33050c13 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst @@ -28,7 +28,7 @@ Install OpenVINO™ Runtime on Linux .. tip:: - The new OpenVINO GenAI package is now available for installation via :doc:`Archives ` and :doc:`PyPI ` distributions. + The new OpenVINO GenAI package is now available for installation via :doc:`Archive ` and :doc:`PyPI ` distributions. If you want to install OpenVINO™ Runtime on Linux, you have the following options: diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 9d0cf35084f..543e1c620e3 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -2,14 +2,14 @@ OpenVINO GenAI API Guide =============================== -This guide shows the essential steps for integrating the OpenVINO GenAI API into your application. +This guide provide the instructions for integrating the OpenVINO GenAI API into your application. The steps below demonstrate how to load a model and pass the input context to receive generated text. The examples use a CPU as the target device, however, the GPU support is also available. Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. Tokenizers are represented as a separate model and run on the CPU using the provided inference capabilities. -Before proceeding, make sure that you have installed the OpenVINO GenAI API using :doc:`PyPI or Archives <../../get-started/install-openvino>`. +Before proceeding, make sure that you have installed the OpenVINO GenAI API using :doc:`PyPI or Archive <../../get-started/install-openvino>` distributions. 1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used for this example: @@ -62,39 +62,43 @@ Streaming Options For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: -.. code-block:: cpp +.. tab-set:: - #include "openvino/genai/llm_pipeline.hpp" - #include + .. tab-item:: C++ - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::genai::LLMPipeline pipe(model_path, "CPU"); + #include "openvino/genai/llm_pipeline.hpp" + #include - auto streamer = [](std::string word) { std::cout << word << std::flush; }; - std::cout << pipe.generate("The Sun is yellow because", streamer); - } + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU"); + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + std::cout << pipe.generate("The Sun is yellow because", streamer); + } You can also create your custom streamer for more sophisticated processing: -.. code-block:: cpp +.. tab-set:: - #include + .. tab-item:: C++ - class CustomStreamer: publict StreamerBase { - public: - void put(int64_t token) {/* decode tokens and do process them*/}; + #include - void end() {/* decode tokens and do process them*/}; - }; + class CustomStreamer: publict StreamerBase { + public: + void put(int64_t token) {/* decode tokens and do process them*/}; - int main(int argc, char* argv[]) { - CustomStreamer custom_streamer; + void end() {/* decode tokens and do process them*/}; + }; - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); - cout << pipe.generate("The Sun is yellow bacause", custom_streamer); - } + int main(int argc, char* argv[]) { + CustomStreamer custom_streamer; + + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + cout << pipe.generate("The Sun is yellow bacause", custom_streamer); + } Chat Scenarios Optimization ############################## @@ -154,79 +158,85 @@ Leverage group beam search decoding and configure generation_config for better t Use group beam search decoding: -.. code-block:: cpp +.. tab-set:: - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + .. tab-item:: C++ - cout << pipe.generate("The Sun is yellow bacause", config); - } + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + cout << pipe.generate("The Sun is yellow bacause", config); + } Specify generation_config to use grouped beam search: -.. code-block:: cpp +.. tab-set:: - int main(int argc, char* argv[]) { - std::string prompt; + .. tab-item:: C++ - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); + int main(int argc, char* argv[]) { + std::string prompt; - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); - auto streamer = [](std::string word) { std::cout << word << std::flush; }; + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; - pipe.start_chat(); - for (size_t i = 0; i < questions.size(); i++) { + auto streamer = [](std::string word) { std::cout << word << std::flush; }; - std::cout << "question:\n"; - cout << prompt << endl; + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { - auto answer = pipe(prompt, config, streamer); - // no need to print answer, streamer will do that + std::cout << "question:\n"; + cout << prompt << endl; + + auto answer = pipe(prompt, config, streamer); + // no need to print answer, streamer will do that + } + pipe.finish_chat(); } - pipe.finish_chat(); - } Comparing with Hugging Face Results ####################################### Compare and analyze results with those generated by Hugging Face models. -.. code-block:: python +.. tab-set:: - from transformers import AutoTokenizer, AutoModelForCausalLM + .. tab-item:: Python - tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + from transformers import AutoTokenizer, AutoModelForCausalLM - max_new_tokens = 32 - prompt = 'table is made of' + tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) - hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) - hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) - print(f'hf_output: {hf_output}') + max_new_tokens = 32 + prompt = 'table is made of' - import sys - sys.path.append('build-Debug/') - import py_generate_pipeline as genai # set more friendly module name + encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) + hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) + hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) + print(f'hf_output: {hf_output}') - pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') - ov_output = pipe(prompt, max_new_tokens=max_new_tokens) - print(f'ov_output: {ov_output}') + import sys + sys.path.append('build-Debug/') + import py_generate_pipeline as genai # set more friendly module name - assert hf_output == ov_output + pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') + ov_output = pipe(prompt, max_new_tokens=max_new_tokens) + print(f'ov_output: {ov_output}') + + assert hf_output == ov_output From 34a538bedd2897fbb24b9149bc5829413fa06dd8 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 6 Jun 2024 13:13:35 +0200 Subject: [PATCH 06/21] fix directives --- .../llm_inference_guide/genai-guide.rst | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 543e1c620e3..d32512466e7 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -62,9 +62,9 @@ Streaming Options For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: -.. tab-set:: +.. tab-item:: C++ - .. tab-item:: C++ + .. code-block:: cpp #include "openvino/genai/llm_pipeline.hpp" #include @@ -79,9 +79,10 @@ For more interactive UIs during generation, streaming of model output tokens is You can also create your custom streamer for more sophisticated processing: -.. tab-set:: - .. tab-item:: C++ +.. tab-item:: C++ + + .. code-block:: cpp #include @@ -158,9 +159,9 @@ Leverage group beam search decoding and configure generation_config for better t Use group beam search decoding: -.. tab-set:: +.. tab-item:: C++ - .. tab-item:: C++ + .. code-block:: cpp int main(int argc, char* argv[]) { std::string model_path = argv[1]; @@ -176,9 +177,9 @@ Use group beam search decoding: Specify generation_config to use grouped beam search: -.. tab-set:: +.. tab-item:: C++ - .. tab-item:: C++ + .. code-block:: cpp int main(int argc, char* argv[]) { std::string prompt; @@ -211,9 +212,9 @@ Comparing with Hugging Face Results Compare and analyze results with those generated by Hugging Face models. -.. tab-set:: +.. tab-item:: Python - .. tab-item:: Python + .. code-block:: python from transformers import AutoTokenizer, AutoModelForCausalLM From 009aa56584dc3326d97d427c8b704967251c7f17 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 6 Jun 2024 13:32:21 +0200 Subject: [PATCH 07/21] add tab set --- .../llm_inference_guide/genai-guide.rst | 157 +++++++++--------- 1 file changed, 83 insertions(+), 74 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index d32512466e7..9114df49019 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -62,44 +62,47 @@ Streaming Options For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: -.. tab-item:: C++ +.. tab-set:: - .. code-block:: cpp + .. tab-item:: C++ - #include "openvino/genai/llm_pipeline.hpp" - #include + .. code-block:: cpp - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::genai::LLMPipeline pipe(model_path, "CPU"); + #include "openvino/genai/llm_pipeline.hpp" + #include - auto streamer = [](std::string word) { std::cout << word << std::flush; }; - std::cout << pipe.generate("The Sun is yellow because", streamer); - } + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU"); + + auto streamer = [](std::string word) { std::cout << word << std::flush; }; + std::cout << pipe.generate("The Sun is yellow because", streamer); + } You can also create your custom streamer for more sophisticated processing: +.. tab-set:: -.. tab-item:: C++ + .. tab-item:: C++ - .. code-block:: cpp + .. code-block:: cpp - #include + #include - class CustomStreamer: publict StreamerBase { - public: - void put(int64_t token) {/* decode tokens and do process them*/}; + class CustomStreamer: publict StreamerBase { + public: + void put(int64_t token) {/* decode tokens and do process them*/}; - void end() {/* decode tokens and do process them*/}; - }; + void end() {/* decode tokens and do process them*/}; + }; - int main(int argc, char* argv[]) { - CustomStreamer custom_streamer; + int main(int argc, char* argv[]) { + CustomStreamer custom_streamer; - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); - cout << pipe.generate("The Sun is yellow bacause", custom_streamer); - } + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + cout << pipe.generate("The Sun is yellow bacause", custom_streamer); + } Chat Scenarios Optimization ############################## @@ -159,85 +162,91 @@ Leverage group beam search decoding and configure generation_config for better t Use group beam search decoding: -.. tab-item:: C++ +.. tab-set:: - .. code-block:: cpp + .. tab-item:: C++ - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + .. code-block:: cpp - cout << pipe.generate("The Sun is yellow bacause", config); - } + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; + + cout << pipe.generate("The Sun is yellow bacause", config); + } Specify generation_config to use grouped beam search: -.. tab-item:: C++ +.. tab-set:: - .. code-block:: cpp + .. tab-item:: C++ - int main(int argc, char* argv[]) { - std::string prompt; + .. code-block:: cpp - std::string model_path = argv[1]; - ov::LLMPipeline pipe(model_path, "CPU"); + int main(int argc, char* argv[]) { + std::string prompt; - ov::GenerationConfig config = pipe.get_generation_config(); - config.max_new_tokens = 256; - config.num_groups = 3; - config.group_size = 5; - config.diversity_penalty = 1.0f; + std::string model_path = argv[1]; + ov::LLMPipeline pipe(model_path, "CPU"); - auto streamer = [](std::string word) { std::cout << word << std::flush; }; + ov::GenerationConfig config = pipe.get_generation_config(); + config.max_new_tokens = 256; + config.num_groups = 3; + config.group_size = 5; + config.diversity_penalty = 1.0f; - pipe.start_chat(); - for (size_t i = 0; i < questions.size(); i++) { + auto streamer = [](std::string word) { std::cout << word << std::flush; }; - std::cout << "question:\n"; - cout << prompt << endl; + pipe.start_chat(); + for (size_t i = 0; i < questions.size(); i++) { - auto answer = pipe(prompt, config, streamer); - // no need to print answer, streamer will do that + std::cout << "question:\n"; + cout << prompt << endl; + + auto answer = pipe(prompt, config, streamer); + // no need to print answer, streamer will do that + } + pipe.finish_chat(); } - pipe.finish_chat(); - } Comparing with Hugging Face Results ####################################### Compare and analyze results with those generated by Hugging Face models. -.. tab-item:: Python +.. tab-set:: - .. code-block:: python + .. tab-item:: Python - from transformers import AutoTokenizer, AutoModelForCausalLM + .. code-block:: python - tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + from transformers import AutoTokenizer, AutoModelForCausalLM - max_new_tokens = 32 - prompt = 'table is made of' + tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") + model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") - encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) - hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) - hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) - print(f'hf_output: {hf_output}') + max_new_tokens = 32 + prompt = 'table is made of' - import sys - sys.path.append('build-Debug/') - import py_generate_pipeline as genai # set more friendly module name + encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False) + hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False) + hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:]) + print(f'hf_output: {hf_output}') - pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') - ov_output = pipe(prompt, max_new_tokens=max_new_tokens) - print(f'ov_output: {ov_output}') + import sys + sys.path.append('build-Debug/') + import py_generate_pipeline as genai # set more friendly module name - assert hf_output == ov_output + pipe = genai.LLMPipeline('text_generation/causal_lm/TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/') + ov_output = pipe(prompt, max_new_tokens=max_new_tokens) + print(f'ov_output: {ov_output}') + + assert hf_output == ov_output From 07a61cd01f255a297c73244b77d689de72f4971f Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 6 Jun 2024 21:08:36 +0200 Subject: [PATCH 08/21] change installation --- .../get-started/install-openvino.rst | 49 +++++++------- .../install-openvino-archive-linux.rst | 4 -- .../install-openvino-genai.rst | 64 +++++++++++++++++++ .../install-openvino-linux.rst | 4 -- .../install-openvino/install-openvino-pip.rst | 20 +----- .../learn-openvino/llm_inference_guide.rst | 11 ++-- .../llm_inference_guide/genai-guide.rst | 49 ++++++++------ .../llm_inference_guide/llm-inference-hf.rst | 2 +- .../llm-inference-native-ov.rst | 2 +- 9 files changed, 129 insertions(+), 76 deletions(-) create mode 100644 docs/articles_en/get-started/install-openvino/install-openvino-genai.rst diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index 14e1443df59..2bdd792f1cc 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -14,7 +14,7 @@ Install OpenVINO™ 2024.2 OpenVINO Runtime on Linux OpenVINO Runtime on Windows OpenVINO Runtime on macOS - + OpenVINO GenAI Flavor .. raw:: html @@ -22,26 +22,7 @@ Install OpenVINO™ 2024.2 -.. tip:: - - The new OpenVINO GenAI API package provides a set of LLM-specific interfaces to facilitate the integration - of language models into applications. This API hides the complexity of the generation process - and significantly minimizes the amount of code needed for the application to work. - Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs - tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. - For a quickstart guide, refer to the :doc:`GenAI API Guide <../learn-openvino/llm_inference_guide/genai-guide>` or the GenAI API reference page. - -.. warning:: - - The OpenVINO™ Development Tools package has been deprecated and removed from the default - installation options. For new projects, the OpenVINO runtime package now includes - all necessary components. - - The OpenVINO Development Tools is still available for older versions of OpenVINO, - as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`. - - -.. tip:: +.. dropdown:: Supported OpenVINO Versions OpenVINO 2024.2, described here, is not a Long-Term-Support version! All currently supported versions are: @@ -52,6 +33,8 @@ Install OpenVINO™ 2024.2 Moreover, different OpenVINO distributions may support slightly different sets of features. Read installation guides for particular distributions for more details. + Refer to the :doc:`OpenVINO Release Policy <../documentation/about-openvino/release-notes-openvino/release-policy>` + to learn more about the release types. .. dropdown:: Distribution Comparison for OpenVINO 2024.2 @@ -66,9 +49,27 @@ Install OpenVINO™ 2024.2 | \* **Of the Linux systems, versions 22.04 and 24.04 include drivers for NPU.** | **For Windows, CPU inference on ARM64 is not supported.** -| **Build OpenVINO from source** -| OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform, - follow the `OpenVINO Build Instructions `__. +.. dropdown:: Effortless GenAI integration with OpenVINO GenAI Flavor + + A new OpenVINO GenAI Flavor streamlines application development by providing + LLM-specific interfaces for easy integration of language models, handling tokenization and + text generation. For installation and usage instructions, proceed to + :doc:`Install OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>` and + :doc:`Run LLMs with OpenVINO GenAI Flavor <../learn-openvino/llm_inference_guide/genai-guide>`. + +.. dropdown:: Deprecation of OpenVINO™ Development Tools Package + + The OpenVINO™ Development Tools package has been deprecated and removed from the default + installation options. For new projects, the OpenVINO runtime package now includes + all necessary components. + + The OpenVINO Development Tools is still available for older versions of OpenVINO, + as well as the current one, from the GitHub repository and PyPI. :doc:`Learn more <../documentation/legacy-features/install-dev-tools>`. + +.. dropdown:: Building OpenVINO from source + + OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform, + follow the `OpenVINO Build Instructions `__. diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst index 79fcc312c16..a991bae6fe0 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-archive-linux.rst @@ -32,10 +32,6 @@ Install OpenVINO™ Runtime on Linux from an Archive File RHEL8 x86_64 V V n/a =================== ===== ===== ===== -.. tip:: - - The new OpenVINO GenAI package is now available for installation via Archive distribution. Learn more in the Installing OpenVINO Runtime section. - .. tab-set:: .. tab-item:: System Requirements diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst new file mode 100644 index 00000000000..f0e58cf06e9 --- /dev/null +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -0,0 +1,64 @@ + +Install OpenVINO™ GenAI Flavor +==================================== + +The new OpenVINO GenAI Flavor is an API designed to hide the complexity of the generation process +and significantly minimize the amount of code needed for the application to work. +Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs +tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. +For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`. + +OpenVINO GenAI Flavor is available for installation via Archive and PyPI distributions. + +Archive Installation +############################### + +Use this command to download an archive file with OpenVINO GenAI Flavor for your system: + +.. tab-set:: + + .. tab-item:: x86_64 + :sync: x86-64 + + .. tab-set:: + + .. tab-item:: Ubuntu 24.04 + :sync: ubuntu-24 + + .. code-block:: sh + + + curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz + tar -xf openvino_2024.1.0.tgz + sudo mv l_openvino_toolkit_ubuntu24_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 + + .. tab-item:: Ubuntu 22.04 + :sync: ubuntu-22 + + .. code-block:: sh + + curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2024.1/linux/l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64.tgz --output openvino_2024.1.0.tgz + tar -xf openvino_2024.1.0.tgz + sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 + + + +For full instructions, refer to the OpenVINO Runtime Archive installation for your system: :doc:`Linux `, :doc:`Windows `, and :doc:`macOS `. + +PyPI Installation +############################### + +Use this command to install OpenVINO GenAI via PyPI: + +.. code-block:: python + + python -m pip install openvino-genai + +Refer to the :doc:`OpenVINO Runtime PyPI installation ` for full instructions. + + + + + + + diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst index 67d33050c13..36df7f39ff8 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-linux.rst @@ -26,10 +26,6 @@ Install OpenVINO™ Runtime on Linux Use Docker Use npm -.. tip:: - - The new OpenVINO GenAI package is now available for installation via :doc:`Archive ` and :doc:`PyPI ` distributions. - If you want to install OpenVINO™ Runtime on Linux, you have the following options: * :doc:`Install OpenVINO using an Archive File ` diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst index c2301056ca5..191d7563f6d 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-pip.rst @@ -18,10 +18,6 @@ Install Intel® Distribution of OpenVINO™ Toolkit from PyPI Repository (all x86_64 / arm64 architectures) * macOS offers support only for CPU inference -.. tip:: - - The new OpenVINO GenAI package is now available for installation via PyPI. Learn more in the Installing OpenVINO Runtime section. - .. tab-set:: .. tab-item:: System Requirements @@ -106,21 +102,9 @@ Step 4. Install the Package Use the following command to install OpenVINO Base or OpenVINO GenAI package: +.. code-block:: python -.. tab-set:: - - .. tab-item:: OpenVINO Base Package - - .. code-block:: python - - python -m pip install openvino - - .. tab-item:: OpenVINO with GenAI - - .. code-block:: python - - python -m pip install openvino-genai - + python -m pip install openvino Step 5. Verify that the Package Is Installed ++++++++++++++++++++++++++++++++++++++++++++ diff --git a/docs/articles_en/learn-openvino/llm_inference_guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide.rst index e7931209b9d..0798254e8e5 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide.rst @@ -13,9 +13,9 @@ Large Language Model Inference Guide :maxdepth: 1 :hidden: - OpenVINO GenAI Guide - LLM Inference with Optimum Intel - LLM Inference with OpenVINO API + Run LLMs with OpenVINO GenAI Flavor + Run LLMs with Optimum Intel + Run LLMs with Base OpenVINO OpenVINO Tokenizers Large Language Models (LLMs) like GPT are transformative deep learning networks capable of a @@ -40,12 +40,13 @@ The advantages of using OpenVINO for LLM deployment: * **Provides stateful model optimization**: models from the Hugging Face Transformers are converted into a stateful form, optimizing inference performance and memory usage in long-running text generation tasks by managing past KV-cache tensors more efficiently internally. This feature is automatically activated for many supported models, while unsupported ones remain stateless. Learn more about the :doc:`Stateful models and State API <../openvino-workflow/running-inference/stateful-models>`. -OpenVINO offers two main paths for Generative AI use cases: +OpenVINO offers three main paths for Generative AI use cases: +* **OpenVINO GenAI Flavor**: use OpenVINO GenAI APIs (Python and C++). * **Hugging Face**: use OpenVINO as a backend for Hugging Face frameworks (transformers, diffusers) through the `Optimum Intel `__ extension. -* **Native OpenVINO**: use OpenVINO native APIs (Python and C++) with +* **Base OpenVINO**: use OpenVINO native APIs (Python and C++) with `custom pipeline code `__. In both cases, the OpenVINO runtime is used for inference, and OpenVINO tools are used for diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 9114df49019..b662121843e 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -1,15 +1,17 @@ -OpenVINO GenAI API Guide -=============================== +Run LLMs with OpenVINO GenAI Flavor +===================================== -This guide provide the instructions for integrating the OpenVINO GenAI API into your application. -The steps below demonstrate how to load a model and pass the input context to receive generated text. +This guide shows how to integrate the OpenVINO GenAI Flavor into your application, covering +model loading and passing the input context to receive generated text. -The examples use a CPU as the target device, however, the GPU support is also available. -Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. -Tokenizers are represented as a separate model and run on the CPU using the provided inference capabilities. +Before proceeding, make sure that you have :doc:`installed the OpenVINO GenAI Flavor <../../get-started/install-openvino/install-openvino-genai>`. -Before proceeding, make sure that you have installed the OpenVINO GenAI API using :doc:`PyPI or Archive <../../get-started/install-openvino>` distributions. +.. note:: + + The examples use a CPU as the target device, however, the GPU support is also available. + Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. + Tokenizers are represented as a separate model and also run on the CPU. 1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used for this example: @@ -19,8 +21,8 @@ Before proceeding, make sure that you have installed the OpenVINO GenAI API usin *Optional*. Optimize the model: -The model is an optimized OpenVINO IR with fp16 precision. For enhanced LLM performance, -it is recommended to use lower precision for model weights, such as int4, and to compress weights +The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, +it is recommended to use lower precision for model weights, such as INT4, and to compress weights using NNCF during model export directly: .. code-block:: python @@ -55,9 +57,9 @@ using NNCF during model export directly: Once the model is exported from Hugging Face Optimum-Intel, it already contains all the necessary information for execution, including the tokenizer/detokenizer and the generation config -ensuring that its results match Hugging Face generation. +ensuring that its results `match those generated by Hugging Face `__. -Streaming Options +Streaming the Output ########################### For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: @@ -101,11 +103,11 @@ You can also create your custom streamer for more sophisticated processing: std::string model_path = argv[1]; ov::LLMPipeline pipe(model_path, "CPU"); - cout << pipe.generate("The Sun is yellow bacause", custom_streamer); + cout << pipe.generate("The Sun is yellow because", custom_streamer); } -Chat Scenarios Optimization -############################## +Optimizing the Chat Scenario +################################ For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs offers optimization benefits. The chat-specific methods **start_chat** and **finish_chat** are used to @@ -155,12 +157,12 @@ mark a conversation session. Simplified Python and C++ examples are provided bel pipe.finish_chat(); } -Optimizing Text Generation with Group Beam Search +Optimizing Generation with Grouped Beam Search ####################################################### -Leverage group beam search decoding and configure generation_config for better text generation quality and efficient batch processing in GenAI applications. +Leverage grouped beam search decoding and configure generation_config for better text generation quality and efficient batch processing in GenAI applications. -Use group beam search decoding: +Use grouped beam search decoding: .. tab-set:: @@ -177,7 +179,7 @@ Use group beam search decoding: config.group_size = 5; config.diversity_penalty = 1.0f; - cout << pipe.generate("The Sun is yellow bacause", config); + cout << pipe.generate("The Sun is yellow because", config); } Specify generation_config to use grouped beam search: @@ -248,7 +250,16 @@ Compare and analyze results with those generated by Hugging Face models. assert hf_output == ov_output +GenAI API +####################################### +OpenVINO GenAI Flavor includes the following API: + +* generation_config +* llm_pipeline +* streamer_base +* tokenizer +* visibility Additional Resources #################### diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst index aa5ad1066b5..a0edebf83d1 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-hf.rst @@ -1,6 +1,6 @@ .. {#llm_inference} -Inference with Hugging Face and Optimum Intel +Run LLMs with Hugging Face and Optimum Intel ===================================================== The steps below show how to load and infer LLMs from Hugging Face using Optimum Intel. diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst index 33b1310fe65..e337ca832a8 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst @@ -1,6 +1,6 @@ .. {#llm_inference_native_ov} -Inference with Native OpenVINO +Run LLMs with Base OpenVINO =============================== To run Generative AI models using native OpenVINO APIs you need to follow regular **Convert -> Optimize -> Deploy** path with a few simplifications. From ea4fde23123b96bcd94242542883fd1df3078e82 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 6 Jun 2024 21:44:21 +0200 Subject: [PATCH 09/21] fix rn policy link --- docs/articles_en/get-started/install-openvino.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/articles_en/get-started/install-openvino.rst b/docs/articles_en/get-started/install-openvino.rst index 2bdd792f1cc..6c6eac4c592 100644 --- a/docs/articles_en/get-started/install-openvino.rst +++ b/docs/articles_en/get-started/install-openvino.rst @@ -33,7 +33,7 @@ Install OpenVINO™ 2024.2 Moreover, different OpenVINO distributions may support slightly different sets of features. Read installation guides for particular distributions for more details. - Refer to the :doc:`OpenVINO Release Policy <../documentation/about-openvino/release-notes-openvino/release-policy>` + Refer to the :doc:`OpenVINO Release Policy <../../../about-openvino/release-notes-openvino/release-policy>` to learn more about the release types. .. dropdown:: Distribution Comparison for OpenVINO 2024.2 From fd2671c5aebdd264410b150eabe352ad2e434b57 Mon Sep 17 00:00:00 2001 From: Karol Blaszczak Date: Fri, 7 Jun 2024 09:11:41 +0200 Subject: [PATCH 10/21] Update install-openvino-genai.rst --- .../install-openvino-genai.rst | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index f0e58cf06e9..adc9cec6715 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -1,19 +1,23 @@ - -Install OpenVINO™ GenAI Flavor +Install OpenVINO™ GenAI ==================================== -The new OpenVINO GenAI Flavor is an API designed to hide the complexity of the generation process -and significantly minimize the amount of code needed for the application to work. -Developers can now provide a model and input context directly to the OpenVINO GenAI, which performs -tokenization of the input text, executes the generation loop on the selected device, and then returns the generated text. +OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of denerative AI models. +It hides the complexity of the generation process and minimizes the amount of code required. +You can now provide a model and input context directly to OpenVINO, which performs tokenization of the +input text, executes the generation loop on the selected device, and returns the generated text. For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`. -OpenVINO GenAI Flavor is available for installation via Archive and PyPI distributions. +To see GenAI in action, check the Jupyter notebooks: +`LLM-powered Chatbot `__ and +`LLM Instruction-following pipeline `__ + +The OpenVINO GenAI flavor is available for installation via Archive and PyPI distributions: Archive Installation ############################### -Use this command to download an archive file with OpenVINO GenAI Flavor for your system: +To install the GenAI flavor of OpenVINO from an archive file, follow the standard installation steps for your system +but instead of using the vanilla package file, download the one with OpenVINO GenAI: .. tab-set:: @@ -41,24 +45,22 @@ Use this command to download an archive file with OpenVINO GenAI Flavor for your tar -xf openvino_2024.1.0.tgz sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 +Here are the full guides: +:doc:`Linux `, +:doc:`Windows `, and +:doc:`macOS `. -For full instructions, refer to the OpenVINO Runtime Archive installation for your system: :doc:`Linux `, :doc:`Windows `, and :doc:`macOS `. - PyPI Installation ############################### -Use this command to install OpenVINO GenAI via PyPI: +To install the GenAI flavor of OpenVINO via PyPI, follow the standard :doc:`installation steps `, +but use the *openvino-genai* package instead of *openvino*: .. code-block:: python python -m pip install openvino-genai -Refer to the :doc:`OpenVINO Runtime PyPI installation ` for full instructions. - - - - From b1a97d63c18344827769f767496cae7af9642b0b Mon Sep 17 00:00:00 2001 From: Karol Blaszczak Date: Fri, 7 Jun 2024 09:49:40 +0200 Subject: [PATCH 11/21] Update genai-guide.rst --- .../llm_inference_guide/genai-guide.rst | 85 ++++++++++--------- 1 file changed, 45 insertions(+), 40 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index b662121843e..8010d549df4 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -1,68 +1,72 @@ - Run LLMs with OpenVINO GenAI Flavor ===================================== -This guide shows how to integrate the OpenVINO GenAI Flavor into your application, covering -model loading and passing the input context to receive generated text. +.. meta:: + :description: Learn how to use the OpenVINO GenAI flavor to execute LLM models. -Before proceeding, make sure that you have :doc:`installed the OpenVINO GenAI Flavor <../../get-started/install-openvino/install-openvino-genai>`. +This guide will show you how to integrate the OpenVINO GenAI flavor into your application, covering +loading a model and passing the input context to receive generated text. note that the vanilla flavor of OpenVINO +will not work with these instructions, make sure to +:doc:`install OpenVINO GenAI <../../get-started/install-openvino/install-openvino-genai>`. .. note:: - The examples use a CPU as the target device, however, the GPU support is also available. - Note that the GPU is running only LLM inference, while token selection and tokenization/detokenization remain on the CPU for efficiency. - Tokenizers are represented as a separate model and also run on the CPU. + The examples use the CPU as the target device, however, the GPU is also supported. + Note that for the LLM pipeline, the GPU is used only for inferenc, while token selection, tokenization, and + detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run + on the CPU. -1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used for this example: +1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example: -.. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" -*Optional*. Optimize the model: + *Optional*. Optimize the model: -The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, -it is recommended to use lower precision for model weights, such as INT4, and to compress weights -using NNCF during model export directly: + The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, + it is recommended to use lower precision for model weights, such as INT4, and to compress weights + using NNCF during model export directly: -.. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code 2. Perform generation using the new GenAI API: -.. tab-set:: + .. tab-set:: - .. tab-item:: Python - :sync: py + .. tab-item:: Python + :sync: py - .. code-block:: python + .. code-block:: python - import openvino_genai as ov_genai - pipe = ov_genai.LLMPipeline(model_path, "CPU") - print(pipe.generate("The Sun is yellow because")) + import openvino_genai as ov_genai + pipe = ov_genai.LLMPipeline(model_path, "CPU") + print(pipe.generate("The Sun is yellow because")) - .. tab-item:: C++ - :sync: cpp + .. tab-item:: C++ + :sync: cpp - .. code-block:: cpp + .. code-block:: cpp - #include "openvino/genai/llm_pipeline.hpp" - #include + #include "openvino/genai/llm_pipeline.hpp" + #include - int main(int argc, char* argv[]) { - std::string model_path = argv[1]; - ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU - std::cout << pipe.generate("The Sun is yellow because"); //input context + int main(int argc, char* argv[]) { + std::string model_path = argv[1]; + ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU + std::cout << pipe.generate("The Sun is yellow because"); //input context -Once the model is exported from Hugging Face Optimum-Intel, it already contains all the necessary -information for execution, including the tokenizer/detokenizer and the generation config -ensuring that its results `match those generated by Hugging Face `__. +Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information +necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that +its results `match those generated by Hugging Face `__. Streaming the Output ########################### -For more interactive UIs during generation, streaming of model output tokens is supported. See the example below, where a lambda function outputs words to the console immediately upon generation: +For more interactive UIs during generation, streaming of model output tokens is supported. See the example +below, where a lambda function outputs words to the console immediately upon generation: .. tab-set:: @@ -110,8 +114,8 @@ Optimizing the Chat Scenario ################################ For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache across inputs -offers optimization benefits. The chat-specific methods **start_chat** and **finish_chat** are used to -mark a conversation session. Simplified Python and C++ examples are provided below: +may prove beneficial. The chat-specific methods **start_chat** and **finish_chat** are used to +mark a conversation session, as you can see in these simple examples: .. tab-set:: @@ -160,7 +164,8 @@ mark a conversation session. Simplified Python and C++ examples are provided bel Optimizing Generation with Grouped Beam Search ####################################################### -Leverage grouped beam search decoding and configure generation_config for better text generation quality and efficient batch processing in GenAI applications. +Leverage grouped beam search decoding and configure generation_config for better text generation +quality and efficient batch processing in GenAI applications. Use grouped beam search decoding: @@ -253,7 +258,7 @@ Compare and analyze results with those generated by Hugging Face models. GenAI API ####################################### -OpenVINO GenAI Flavor includes the following API: +OpenVINO GenAI Flavor includes the following API methods: * generation_config * llm_pipeline From a4303f876a16b8a7f2354e2570184daa48bf625f Mon Sep 17 00:00:00 2001 From: Karol Blaszczak Date: Fri, 7 Jun 2024 09:51:31 +0200 Subject: [PATCH 12/21] just a stupid letter :) --- .../get-started/install-openvino/install-openvino-genai.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index adc9cec6715..42fed65c0ae 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -1,7 +1,7 @@ Install OpenVINO™ GenAI ==================================== -OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of denerative AI models. +OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of generative AI models. It hides the complexity of the generation process and minimizes the amount of code required. You can now provide a model and input context directly to OpenVINO, which performs tokenization of the input text, executes the generation loop on the selected device, and returns the generated text. From 2ab5beb98e807db8c12e550f13fe29314821f209 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Fri, 7 Jun 2024 09:57:37 +0200 Subject: [PATCH 13/21] rm link to header --- .../learn-openvino/llm_inference_guide/genai-guide.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 8010d549df4..57abbfceb7b 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -5,14 +5,14 @@ Run LLMs with OpenVINO GenAI Flavor :description: Learn how to use the OpenVINO GenAI flavor to execute LLM models. This guide will show you how to integrate the OpenVINO GenAI flavor into your application, covering -loading a model and passing the input context to receive generated text. note that the vanilla flavor of OpenVINO +loading a model and passing the input context to receive generated text. Note that the vanilla flavor of OpenVINO will not work with these instructions, make sure to :doc:`install OpenVINO GenAI <../../get-started/install-openvino/install-openvino-genai>`. .. note:: The examples use the CPU as the target device, however, the GPU is also supported. - Note that for the LLM pipeline, the GPU is used only for inferenc, while token selection, tokenization, and + Note that for the LLM pipeline, the GPU is used only for inference, while token selection, tokenization, and detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run on the CPU. @@ -58,9 +58,9 @@ will not work with these instructions, make sure to ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU std::cout << pipe.generate("The Sun is yellow because"); //input context -Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information +Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that -its results `match those generated by Hugging Face `__. +its results match those generated by Hugging Face comparing-with-hugging-face-results. Streaming the Output ########################### From 6a8719ad0aec95cd05bd33b5989a087178d46ac1 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Fri, 7 Jun 2024 09:58:23 +0200 Subject: [PATCH 14/21] rm link to header --- .../learn-openvino/llm_inference_guide/genai-guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 57abbfceb7b..02ff5613710 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -60,7 +60,7 @@ will not work with these instructions, make sure to Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that -its results match those generated by Hugging Face comparing-with-hugging-face-results. +its results match those generated by Hugging Face. Streaming the Output ########################### From cf0b68fd8829625ab0bf8098ec51e593ed82cf06 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Fri, 7 Jun 2024 11:06:05 +0200 Subject: [PATCH 15/21] try different space --- .../llm_inference_guide/genai-guide.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 02ff5613710..b9840838b8a 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -18,19 +18,19 @@ will not work with these instructions, make sure to 1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example: - .. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" - *Optional*. Optimize the model: + *Optional*. Optimize the model: - The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, - it is recommended to use lower precision for model weights, such as INT4, and to compress weights - using NNCF during model export directly: + The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, + it is recommended to use lower precision for model weights, such as INT4, and to compress weights + using NNCF during model export directly: - .. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code 2. Perform generation using the new GenAI API: From 042a39b1e6c7c9b4799bcfa4033cf36bf4e7c3ec Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Fri, 7 Jun 2024 11:18:41 +0200 Subject: [PATCH 16/21] fix spaces --- .../llm_inference_guide/genai-guide.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index b9840838b8a..3e53085b97d 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -16,21 +16,21 @@ will not work with these instructions, make sure to detokenization remain on the CPU, for efficiency. Tokenizers are represented as a separate model and also run on the CPU. -1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example: +1. Export an LLM model via Hugging Face Optimum-Intel. A chat-tuned TinyLlama model is used in this example: - .. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format fp16 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0" - *Optional*. Optimize the model: + *Optional*. Optimize the model: - The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, - it is recommended to use lower precision for model weights, such as INT4, and to compress weights - using NNCF during model export directly: + The model is an optimized OpenVINO IR with FP16 precision. For enhanced LLM performance, + it is recommended to use lower precision for model weights, such as INT4, and to compress weights + using NNCF during model export directly: - .. code-block:: python + .. code-block:: python - optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code + optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code 2. Perform generation using the new GenAI API: From 443e687dfc710da46d01ff6a997db492bf9cbfb6 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Mon, 10 Jun 2024 23:09:03 +0200 Subject: [PATCH 17/21] table draft --- .../install-openvino-genai.rst | 90 ++++++++++++++++++- .../llm_inference_guide/genai-guide.rst | 17 ++-- .../llm-inference-native-ov.rst | 2 - 3 files changed, 98 insertions(+), 11 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index 42fed65c0ae..7af9ee95921 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -1,4 +1,4 @@ -Install OpenVINO™ GenAI +Install OpenVINO™ GenAI ==================================== OpenVINO GenAI is a new flavor of OpenVINO, aiming to simplify running inference of generative AI models. @@ -7,8 +7,8 @@ You can now provide a model and input context directly to OpenVINO, which perfor input text, executes the generation loop on the selected device, and returns the generated text. For a quickstart guide, refer to the :doc:`GenAI API Guide <../../learn-openvino/llm_inference_guide/genai-guide>`. -To see GenAI in action, check the Jupyter notebooks: -`LLM-powered Chatbot `__ and +To see GenAI in action, check the Jupyter notebooks: +`LLM-powered Chatbot `__ and `LLM Instruction-following pipeline `__ The OpenVINO GenAI flavor is available for installation via Archive and PyPI distributions: @@ -46,10 +46,21 @@ but instead of using the vanilla package file, download the one with OpenVINO Ge sudo mv l_openvino_toolkit_ubuntu22_2024.1.0.15008.f4afc983258_x86_64 /opt/intel/openvino_2024.1.0 Here are the full guides: -:doc:`Linux `, +:doc:`Linux `, :doc:`Windows `, and :doc:`macOS `. +If OpenVINO GenAI is installed via archive distribution or built from source, you will need to install additional python dependencies (for example, `optimum-cli` for simplified model downloading and exporting): + +.. code-block:: sh + + # (Optional) Clone OpenVINO GenAI repository + git clone --recursive https://github.com/openvinotoolkit/openvino.genai.git + cd openvino.genai + # Install python dependencies + python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release + python -m pip install --upgrade-strategy eager -r ./samples/cpp/requirements.txt + PyPI Installation ############################### @@ -62,5 +73,76 @@ but use the *openvino-genai* package instead of *openvino*: python -m pip install openvino-genai +Supported Models +####################################### + +.. list-table:: + :widths: 20 25 55 + :header-rows: 1 + + * - Architecture + - Models + - Example Hugging Face Models + * - ``ChatGLMModel`` + - ChatGLM + - `THUDM/chatglm2-6b `__ + `THUDM/chatglm3-6b `__ + * - ``GemmaForCausalLM`` + - Gemma + - `google/gemma-2b-it `__ + * - ``GPTNeoXForCausalLM`` + - Dolly + RedPajama + - `databricks/dolly-v2-3b `__ + `ikala/redpajama-3b-chat `__ + * - ``LlamaForCausalLM`` + - Llama 2 + OpenLLaMA + TinyLlama + - `meta-llama/Llama-2-13b-chat-hf `__ + `meta-llama/Llama-2-13b-hf `__ + `meta-llama/Llama-2-7b-chat-hf `__ + `meta-llama/Llama-2-7b-hf `__ + `meta-llama/Llama-2-70b-chat-hf `__ + `meta-llama/Llama-2-70b-hf `__ + `microsoft/Llama2-7b-WhoIsHarryPotter `__ + `openlm-research/open_llama_13b `__ + `openlm-research/open_llama_3b `__ + `openlm-research/open_llama_3b_v2 `__ + `openlm-research/open_llama_7b `__ + `openlm-research/open_llama_7b_v2 `__ + `TinyLlama/TinyLlama-1.1B-Chat-v1.0 `__ + * - ``MistralForCausalLM`` + - Mistral + Notus + Zephyr + - `mistralai/Mistral-7B-v0.1 `__ + `argilla/notus-7b-v1 `__ + `HuggingFaceH4/zephyr-7b-beta `__ + * - ``PhiForCausalLM`` + - Phi + - `microsoft/phi-2 `__ + `microsoft/phi-1_5 `__ + * - ``QWenLMHeadModel`` + - Qwen + - `Qwen/Qwen-7B-Chat `__ + `Qwen/Qwen-7B-Chat-Int4 `__ + `Qwen/Qwen1.5-7B-Chat `__ + `Qwen/Qwen1.5-7B-Chat-GPTQ-Int4 `__ + +The pipeline can work with other similar topologies produced by Optimum Intel with the same model +signature. After conversion, the model must have the following inputs: + +* ``input_ids`` contains the tokens. +* ``attention_mask`` is populated with 1s. +* ``beam_idx`` is used to select beams. +* ``position_ids`` (optional) encodes the position of the currently generating token in the sequence +and a single ``logits`` output. + +.. note:: + + Models should belong to the same family and have the same tokenizers. + Some models may require access request submission on the Hugging Face page to be downloaded. + diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 3e53085b97d..b8c7a4b0a5f 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -58,6 +58,10 @@ will not work with these instructions, make sure to ov::genai::LLMPipeline pipe(model_path, "CPU");//target device is CPU std::cout << pipe.generate("The Sun is yellow because"); //input context +The `LLMPipeline` is the main object used for decoding. You can construct it directly from the +folder with the converted model. It will automatically load the main model, tokenizer, detokenizer, +and the default generation configuration. + Once the model is exported from Hugging Face Optimum-Intel, it already contains all the information necessary for execution, including the tokenizer/detokenizer and the generation config, ensuring that its results match those generated by Hugging Face. @@ -260,12 +264,15 @@ GenAI API OpenVINO GenAI Flavor includes the following API methods: -* generation_config -* llm_pipeline -* streamer_base -* tokenizer +* generation_config - defines a configuration class for text generation, +enabling customization of the generation process such as the maximum length of the +generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding +strategy (greedy, beam search, or multinomial sampling). +* llm_pipeline - provides classes and utilities for text generation, +including a pipeline for processing inputs, generating text, and managing outputs with configurable options. +* streamer_base - an abstract base class for creating streamers. +* tokenizer - the tokenizer class for text encoding and decoding. * visibility - Additional Resources #################### diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst index e337ca832a8..12850aa5ab3 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/llm-inference-native-ov.rst @@ -21,7 +21,6 @@ Inference code that uses native API cannot benefit from Hugging Face pipelines. To write such pipelines, you can follow the examples provided as part of OpenVINO: -* `Text generation C++ samples that support most popular models like LLaMA 2 `__ * `OpenVINO Latent Consistency Model C++ image generation pipeline `__ * `OpenVINO Stable Diffusion (with LoRA) C++ image generation pipeline `__ @@ -174,7 +173,6 @@ This step is essential for interpreting the model's output. Additional Resources #################### -* `Text generation C++ samples that support most popular models like LLaMA 2 `__ * `OpenVINO GenAI Repo `__ * `OpenVINO Tokenizers `__ * `Neural Network Compression Framework `__ From 37d7b717c145593046dcdbdca2072581b0397bbc Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Mon, 10 Jun 2024 23:23:35 +0200 Subject: [PATCH 18/21] fix format --- .../install-openvino-genai.rst | 74 +++++++++---------- .../llm_inference_guide/genai-guide.rst | 1 + 2 files changed, 35 insertions(+), 40 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index 7af9ee95921..ff1125e1dee 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -84,51 +84,45 @@ Supported Models - Models - Example Hugging Face Models * - ``ChatGLMModel`` - - ChatGLM - - `THUDM/chatglm2-6b `__ - `THUDM/chatglm3-6b `__ + - | ChatGLM + - | `THUDM/chatglm2-6b `__ + | `THUDM/chatglm3-6b `__ * - ``GemmaForCausalLM`` - - Gemma - - `google/gemma-2b-it `__ + - | Gemma + - | `google/gemma-2b-it `__ * - ``GPTNeoXForCausalLM`` - - Dolly - RedPajama - - `databricks/dolly-v2-3b `__ - `ikala/redpajama-3b-chat `__ + - | Dolly + | RedPajama + - | `databricks/dolly-v2-3b `__ + | `ikala/redpajama-3b-chat `__ * - ``LlamaForCausalLM`` - - Llama 2 - OpenLLaMA - TinyLlama - - `meta-llama/Llama-2-13b-chat-hf `__ - `meta-llama/Llama-2-13b-hf `__ - `meta-llama/Llama-2-7b-chat-hf `__ - `meta-llama/Llama-2-7b-hf `__ - `meta-llama/Llama-2-70b-chat-hf `__ - `meta-llama/Llama-2-70b-hf `__ - `microsoft/Llama2-7b-WhoIsHarryPotter `__ - `openlm-research/open_llama_13b `__ - `openlm-research/open_llama_3b `__ - `openlm-research/open_llama_3b_v2 `__ - `openlm-research/open_llama_7b `__ - `openlm-research/open_llama_7b_v2 `__ - `TinyLlama/TinyLlama-1.1B-Chat-v1.0 `__ + - | Llama 2 + | OpenLLaMA + | TinyLlama + - | `meta-llama/Llama-2-13b-chat-hf `__ + | `meta-llama/Llama-2-13b-hf `__ + | `meta-llama/Llama-2-7b-chat-hf `__ + | `meta-llama/Llama-2-7b-hf `__ + | `meta-llama/Llama-2-70b-chat-hf `__ + | `meta-llama/Llama-2-70b-hf `__ + | `microsoft/Llama2-7b-WhoIsHarryPotter `__ + | `openlm-research/open_llama_13b `__ + | `openlm-research/open_llama_3b `__ + | `openlm-research/open_llama_3b_v2 `__ + | `openlm-research/open_llama_7b `__ + | `openlm-research/open_llama_7b_v2 `__ + | `TinyLlama/TinyLlama-1.1B-Chat-v1.0 `__ * - ``MistralForCausalLM`` - - Mistral - Notus - Zephyr - - `mistralai/Mistral-7B-v0.1 `__ - `argilla/notus-7b-v1 `__ - `HuggingFaceH4/zephyr-7b-beta `__ + - | Mistral + | Notus + | Zephyr + - | `mistralai/Mistral-7B-v0.1 `__ + | `argilla/notus-7b-v1 `__ + | `HuggingFaceH4/zephyr-7b-beta `__ * - ``PhiForCausalLM`` - - Phi - - `microsoft/phi-2 `__ - `microsoft/phi-1_5 `__ - * - ``QWenLMHeadModel`` - - Qwen - - `Qwen/Qwen-7B-Chat `__ - `Qwen/Qwen-7B-Chat-Int4 `__ - `Qwen/Qwen1.5-7B-Chat `__ - `Qwen/Qwen1.5-7B-Chat-GPTQ-Int4 `__ + - | Phi + - | `microsoft/phi-2 `__ + The pipeline can work with other similar topologies produced by Optimum Intel with the same model signature. After conversion, the model must have the following inputs: diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index b8c7a4b0a5f..0cf61b4e669 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -273,6 +273,7 @@ including a pipeline for processing inputs, generating text, and managing output * streamer_base - an abstract base class for creating streamers. * tokenizer - the tokenizer class for text encoding and decoding. * visibility + Additional Resources #################### From d8d1b1248a11e52405517a527383e37983e448f4 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Tue, 11 Jun 2024 12:25:34 +0200 Subject: [PATCH 19/21] apply comments --- .../install-openvino-genai.rst | 68 ------------------- .../learn-openvino/llm_inference_guide.rst | 2 +- .../llm_inference_guide/genai-guide.rst | 10 ++- 3 files changed, 9 insertions(+), 71 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index ff1125e1dee..f651ba904a9 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -72,71 +72,3 @@ but use the *openvino-genai* package instead of *openvino*: python -m pip install openvino-genai - -Supported Models -####################################### - -.. list-table:: - :widths: 20 25 55 - :header-rows: 1 - - * - Architecture - - Models - - Example Hugging Face Models - * - ``ChatGLMModel`` - - | ChatGLM - - | `THUDM/chatglm2-6b `__ - | `THUDM/chatglm3-6b `__ - * - ``GemmaForCausalLM`` - - | Gemma - - | `google/gemma-2b-it `__ - * - ``GPTNeoXForCausalLM`` - - | Dolly - | RedPajama - - | `databricks/dolly-v2-3b `__ - | `ikala/redpajama-3b-chat `__ - * - ``LlamaForCausalLM`` - - | Llama 2 - | OpenLLaMA - | TinyLlama - - | `meta-llama/Llama-2-13b-chat-hf `__ - | `meta-llama/Llama-2-13b-hf `__ - | `meta-llama/Llama-2-7b-chat-hf `__ - | `meta-llama/Llama-2-7b-hf `__ - | `meta-llama/Llama-2-70b-chat-hf `__ - | `meta-llama/Llama-2-70b-hf `__ - | `microsoft/Llama2-7b-WhoIsHarryPotter `__ - | `openlm-research/open_llama_13b `__ - | `openlm-research/open_llama_3b `__ - | `openlm-research/open_llama_3b_v2 `__ - | `openlm-research/open_llama_7b `__ - | `openlm-research/open_llama_7b_v2 `__ - | `TinyLlama/TinyLlama-1.1B-Chat-v1.0 `__ - * - ``MistralForCausalLM`` - - | Mistral - | Notus - | Zephyr - - | `mistralai/Mistral-7B-v0.1 `__ - | `argilla/notus-7b-v1 `__ - | `HuggingFaceH4/zephyr-7b-beta `__ - * - ``PhiForCausalLM`` - - | Phi - - | `microsoft/phi-2 `__ - - -The pipeline can work with other similar topologies produced by Optimum Intel with the same model -signature. After conversion, the model must have the following inputs: - -* ``input_ids`` contains the tokens. -* ``attention_mask`` is populated with 1s. -* ``beam_idx`` is used to select beams. -* ``position_ids`` (optional) encodes the position of the currently generating token in the sequence -and a single ``logits`` output. - -.. note:: - - Models should belong to the same family and have the same tokenizers. - Some models may require access request submission on the Hugging Face page to be downloaded. - - - diff --git a/docs/articles_en/learn-openvino/llm_inference_guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide.rst index 0798254e8e5..bb8ae80ee3b 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide.rst @@ -13,8 +13,8 @@ Large Language Model Inference Guide :maxdepth: 1 :hidden: - Run LLMs with OpenVINO GenAI Flavor Run LLMs with Optimum Intel + Run LLMs with OpenVINO GenAI Flavor Run LLMs with Base OpenVINO OpenVINO Tokenizers diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 0cf61b4e669..5af6047bb08 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -262,17 +262,23 @@ Compare and analyze results with those generated by Hugging Face models. GenAI API ####################################### -OpenVINO GenAI Flavor includes the following API methods: +OpenVINO GenAI Flavor includes the following API: * generation_config - defines a configuration class for text generation, enabling customization of the generation process such as the maximum length of the generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding strategy (greedy, beam search, or multinomial sampling). + * llm_pipeline - provides classes and utilities for text generation, including a pipeline for processing inputs, generating text, and managing outputs with configurable options. + * streamer_base - an abstract base class for creating streamers. + * tokenizer - the tokenizer class for text encoding and decoding. -* visibility + +* visibility - controls the visibility of the GenAI library. + +Learn more about API in the `GenAI repository `__. Additional Resources #################### From 5c5d864e2d8f16aa7f89ffa03c9e7d847ccf40ed Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Tue, 11 Jun 2024 13:15:39 +0200 Subject: [PATCH 20/21] fix list --- .../learn-openvino/llm_inference_guide/genai-guide.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst index 5af6047bb08..b3088dc3aa7 100644 --- a/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst +++ b/docs/articles_en/learn-openvino/llm_inference_guide/genai-guide.rst @@ -264,13 +264,9 @@ GenAI API OpenVINO GenAI Flavor includes the following API: -* generation_config - defines a configuration class for text generation, -enabling customization of the generation process such as the maximum length of the -generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding -strategy (greedy, beam search, or multinomial sampling). +* generation_config - defines a configuration class for text generation, enabling customization of the generation process such as the maximum length of the generated text, whether to ignore end-of-sentence tokens, and the specifics of the decoding strategy (greedy, beam search, or multinomial sampling). -* llm_pipeline - provides classes and utilities for text generation, -including a pipeline for processing inputs, generating text, and managing outputs with configurable options. +* llm_pipeline - provides classes and utilities for text generation, including a pipeline for processing inputs, generating text, and managing outputs with configurable options. * streamer_base - an abstract base class for creating streamers. From 4ca9fa58b6599da7b13f053e58cdcdce39f5b209 Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Tue, 11 Jun 2024 14:48:37 +0200 Subject: [PATCH 21/21] remove note --- .../install-openvino/install-openvino-genai.rst | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst index f651ba904a9..5927d11b18e 100644 --- a/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst +++ b/docs/articles_en/get-started/install-openvino/install-openvino-genai.rst @@ -50,18 +50,6 @@ Here are the full guides: :doc:`Windows `, and :doc:`macOS `. -If OpenVINO GenAI is installed via archive distribution or built from source, you will need to install additional python dependencies (for example, `optimum-cli` for simplified model downloading and exporting): - -.. code-block:: sh - - # (Optional) Clone OpenVINO GenAI repository - git clone --recursive https://github.com/openvinotoolkit/openvino.genai.git - cd openvino.genai - # Install python dependencies - python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release - python -m pip install --upgrade-strategy eager -r ./samples/cpp/requirements.txt - - PyPI Installation ###############################