diff --git a/docs/articles_en/openvino_workflow/gen_ai.rst b/docs/articles_en/openvino_workflow/gen_ai.rst index 538530603b8..2d2d0383f31 100644 --- a/docs/articles_en/openvino_workflow/gen_ai.rst +++ b/docs/articles_en/openvino_workflow/gen_ai.rst @@ -6,31 +6,31 @@ Optimize and Deploy Generative AI Models Generative AI is an innovative technique that creates new data, such as text, images, video, or audio, using neural networks. OpenVINO accelerates Generative AI use cases as they mostly rely on model inference, allowing for faster development and better performance. When it comes to generative models, OpenVINO supports: -* Conversion, optimization and inference for text, image and audio generative models, for example, Llama 2, MPT, OPT, Stable Diffusion, Stable Diffusion XL, etc. -* Int8 weight compression for text generation models. -* Storage format reduction (fp16 precision for non-compressed models and int8 for compressed models). -* Inference on CPU and GPU platforms, including integrated Intel® Processor Graphics, discrete Intel® Arc™ A-Series Graphics, and discrete Intel® Data Center GPU Flex Series. - +* Conversion, optimization and inference for text, image and audio generative models, for example, Llama 2, MPT, OPT, Stable Diffusion, Stable Diffusion XL, etc. +* Int8 weight compression for text generation models. +* Storage format reduction (fp16 precision for non-compressed models and int8 for compressed models). +* Inference on CPU and GPU platforms, including integrated Intel® Processor Graphics, discrete Intel® Arc™ A-Series Graphics, and discrete Intel® Data Center GPU Flex Series. + OpenVINO offers two main paths for Generative AI use cases: * Using OpenVINO as a backend for Hugging Face frameworks (transformers, diffusers) through the `Optimum Intel `__ extension. -* Using OpenVINO native APIs (Python and C++) with custom pipeline code. +* Using OpenVINO native APIs (Python and C++) with custom pipeline code. + - In both cases, OpenVINO runtime and tools are used, the difference is mostly in the preferred API and the final solution's footprint. Native APIs enable the use of generative models in C++ applications, ensure minimal runtime dependencies, and minimize application footprint. The Native APIs approach requires the implementation of glue code (generation loop, text tokenization, or scheduler functions), which is hidden within Hugging Face libraries for a better developer experience. -It is recommended to start with Hugging Face frameworks. Experiment with different models and scenarios to find your fit, and then consider converting to OpenVINO native APIs based on your specific requirements. +It is recommended to start with Hugging Face frameworks. Experiment with different models and scenarios to find your fit, and then consider converting to OpenVINO native APIs based on your specific requirements. -Optimum Intel provides interfaces that enable model optimization (weight compression) using `Neural Network Compression Framework (NNCF) `__, and export models to the OpenVINO model format for use in native API applications. +Optimum Intel provides interfaces that enable model optimization (weight compression) using `Neural Network Compression Framework (NNCF) `__, and export models to the OpenVINO model format for use in native API applications. -The table below summarizes the differences between Hugging Face and Native APIs approaches. +The table below summarizes the differences between Hugging Face and Native APIs approaches. .. list-table:: :widths: 20 25 55 :header-rows: 1 - * - + * - - Hugging Face through OpenVINO - OpenVINO Native API * - Model support @@ -59,7 +59,7 @@ The table below summarizes the differences between Hugging Face and Native APIs - Best -Running Generative AI Models using Hugging Face Optimum Intel +Running Generative AI Models using Hugging Face Optimum Intel ############################################################## Prerequisites @@ -135,9 +135,17 @@ The optimized model can be saved as usual with a call to ``save_pretrained()``. .. note:: - OpenVINO also supports 4-bit models from Hugging Face `Transformers `__ library optimized + OpenVINO also supports 4-bit models from Hugging Face `Transformers `__ library optimized with `GPTQ `__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it. +Another optimization that is applied by default when using ``OVModelForCausalLM`` class is transformation of the model to a stateful form. +This transformation further improves inference performance and decreases amount of allocated runtime memory in long running text generation scenarious. +It is achieved by hiding inputs and outputs of the model that represent past KV-cache tensors, and handling them inside the model in a more efficient way. +This feature is activated automatically for a wide range of supported text generation models, keeping not supported models in a regular, stateless form. + +Model usage are identical for stateful and stateless models as long as Optimum-Intel API is used because KV-cache handling is an internal detail of the text-generation API of Transformers library. +But a form of a model matterns in case when exported from Optimum-Intel OpenVINO model IR is used in an application implemented with native OpenVINO API, because stateful and stateless models have different number of inputs and outputs. +Please refer to a dedicated section of this document below for more information about using native OpenVINO API. Below are some examples of using Optimum-Intel for model conversion and inference: @@ -164,31 +172,31 @@ Low-rank Adaptation (LoRA) is a popular method to tune Generative AI models to a Now the model can be converted to OpenVINO using Optimum Intel Python API or CLI interfaces mentioned above. -Running Generative AI Models using Native OpenVINO APIs +Running Generative AI Models using Native OpenVINO APIs ######################################################## -To run Generative AI models using native OpenVINO APIs you need to follow regular **Сonvert -> Optimize -> Deploy** path with a few simplifications. +To run Generative AI models using native OpenVINO APIs you need to follow regular **Сonvert -> Optimize -> Deploy** path with a few simplifications. -To convert model from Hugging Face you can use Optimum-Intel export feature that allows to export model in OpenVINO format without invoking conversion API and tools directly, as it is shown above. In this case, the conversion process is a bit more simplified. You can still use a regular conversion path if model comes from outside of Hugging Face ecosystem, i.e., in source framework format (PyTorch, etc.) +To convert the Hugging Face model, the recommended way is to use Optimum-Intel export feature that allows to export model in OpenVINO format without invoking conversion API and tools directly, as it is shown above. +In this case, the conversion process is significantly simplified because Optimum-Intel provides necessary conversion parameters which in many cases model-specific and require knowlege of a lot of model input properties. +Moreover, Optimum-Intel applies several model optimization like weight compression and using stateful form by default that further similifies model exporting flow. +You can still use a regular conversion path if model comes from outside of Hugging Face ecosystem, i.e., in source framework format (PyTorch, TensorFlow etc.) Model optimization can be performed within Hugging Face or directly using NNCF as described in the :doc:`weight compression guide `. Inference code that uses native API cannot benefit from Hugging Face pipelines. You need to write your custom code or take it from the available examples. Below are some examples of popular Generative AI scenarios: -* In case of LLMs for text generation, you need to handle tokenization, inference and token selection loop, and de-tokenization. If token selection involves beam search, it also needs to be written. -* For image generation models, you need to make a pipeline that includes several model inferences: inference for source (e.g., text) encoder models, inference loop for diffusion process and inference for decoding part. Scheduler code is also required. - -To write such pipelines, you can follow the examples provided as part of OpenVINO: - -* `llama2.openvino `__ -* `LLM optimization by custom operation embedding for OpenVINO `__ -* `C++ Implementation of Stable Diffusion `__ +* In case of LLMs for text generation, you need to handle tokenization, inference and token sampling, and de-tokenization. If token sampling involves beam search, it also needs to be written. This is covered in details by `C++ Text Generation Samples `__` +* For image generation models, you need to make a pipeline that includes several model inferences: inference for source (e.g., text) encoder models, inference loop for diffusion process and inference for decoding part. Scheduler code is also required. `C++ Implementation of Stable Diffusion `__ is a good reference point. Additional Resources -############################ +##################### -* `Optimum Intel documentation `_ +* `Optimum Intel documentation `__ * :doc:`LLM Weight Compression ` -* `Neural Network Compression Framework `_ - +* `Neural Network Compression Framework `__ +* `GenAI Pipeline Repository `__ +* `OpenVINO Tokenizers `__ +* `Stateful Models Low-Level Details ` +* `Working with Textual Data `