[DOCS] LLMS docs update (#23092)
### Details: http://openvino-doc.iotg.sclab.intel.com/tsavina-genai/native_vs_hugging_face_api.html doc version ### Tickets: - *ticket-id*
This commit is contained in:
Tatiana Savina
GitHub
parent
221847de5b
commit
e70e59b99e
|
|
@ -16,7 +16,7 @@ Learn OpenVINO
|
|||
|
||||
Interactive Tutorials (Python) <tutorials>
|
||||
Sample Applications (Python & C++) <openvino_docs_OV_UG_Samples_Overview>
|
||||
Generative AI Optimization and Deployment <gen_ai_guide>
|
||||
Large Language Models Inference Guide <native_vs_hugging_face_api>
|
||||
|
||||
|
||||
This section will help you get a hands-on experience with OpenVINO even if you are just starting
|
||||
|
|
@ -30,5 +30,5 @@ as well as an experienced user.
|
|||
| :doc:`OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
| The OpenVINO samples (Python and C++) are simple console applications that show how to use specific OpenVINO API features. They can assist you in executing tasks such as loading a model, running inference, querying particular device capabilities, etc.
|
||||
|
||||
| :doc:`Optimize and Deploy Generative AI Models <gen_ai_guide>`
|
||||
| :doc:`Large Language Models in OpenVINO <native_vs_hugging_face_api>`
|
||||
| Detailed information on how OpenVINO accelerates Generative AI use cases and what models it supports. This tutorial provides instructions for running Generative AI models using Hugging Face Optimum Intel and Native OpenVINO APIs.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,249 @@
|
|||
.. {#llm_inference}
|
||||
|
||||
LLM Inference with Hugging Face and Optimum Intel
|
||||
=====================================================
|
||||
|
||||
The steps below show how to load and infer LLMs from Hugging Face using Optimum Intel.
|
||||
They also show how to convert models into OpenVINO IR format so they can be optimized
|
||||
by NNCF and used with other OpenVINO tools.
|
||||
|
||||
Prerequisites
|
||||
############################################################
|
||||
|
||||
* Create a Python environment by following the instructions on the :doc:`Install OpenVINO PIP <openvino_docs_install_guides_overview>` page.
|
||||
* Install the necessary dependencies for Optimum Intel:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
pip install optimum[openvino,nncf]
|
||||
|
||||
|
||||
Loading a Hugging Face Model to Optimum Intel
|
||||
############################################################
|
||||
|
||||
To start using OpenVINO as a backend for Hugging Face, change the original Hugging Face code in two places:
|
||||
|
||||
.. code-block:: diff
|
||||
|
||||
-from transformers import AutoModelForCausalLM
|
||||
+from optimum.intel import OVModelForCausalLM
|
||||
|
||||
model_id = "meta-llama/Llama-2-7b-chat-hf"
|
||||
-model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
+model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||||
|
||||
|
||||
Instead of using ``AutoModelForCasualLM`` from the Hugging Face transformers library,
|
||||
switch to ``OVModelForCasualLM`` from the optimum.intel library. This change enables
|
||||
you to use OpenVINO's optimization features. You may also use other AutoModel types,
|
||||
such as ``OVModelForSeq2SeqLM``, though this guide will focus on CausalLM.
|
||||
|
||||
By setting the parameter ``export=True``, the model is converted to OpenVINO IR format on the fly.
|
||||
|
||||
After that, you can call ``save_pretrained()`` method to save model to the folder in the OpenVINO
|
||||
Intermediate Representation and use it further.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model.save_pretrained("ov_model")
|
||||
|
||||
This will create a new folder called `ov_model` with the LLM in OpenVINO IR format inside.
|
||||
You can change the folder and provide another model directory instead of `ov_model`.
|
||||
|
||||
Once the model is saved, you can load it with the following command:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained("ov_model")
|
||||
|
||||
Converting a Hugging Face Model to OpenVINO IR
|
||||
############################################################
|
||||
|
||||
The optimum-cli tool allows you to convert models from Hugging Face to
|
||||
the OpenVINO IR format:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
optimum-cli export openvino --model <MODEL_NAME> <NEW_MODEL_NAME>
|
||||
|
||||
If you want to convert the `Llama 2` model from Hugging Face to an OpenVINO IR
|
||||
model and name it `ov_llama_2`, the command would look like this:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf ov_llama_2
|
||||
|
||||
In this case, you can load the converted model in OpenVINO representation directly from the disk:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model_id = "llama_openvino"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id)
|
||||
|
||||
|
||||
By default, inference will run on CPU. To select a different inference device, for example, GPU,
|
||||
add ``device="GPU"`` to the ``from_pretrained()`` call. To switch to a different device after
|
||||
the model has been loaded, use the ``.to()`` method. The device naming convention is the same
|
||||
as in OpenVINO native API:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model.to("GPU")
|
||||
|
||||
|
||||
Optimum-Intel API also provides out-of-the-box model optimization through weight compression
|
||||
using NNCF which substantially reduces the model footprint and inference latency:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True, load_in_8bit=True)
|
||||
|
||||
|
||||
Weight compression is applied by default to models larger than one billion parameters and is
|
||||
also available for CLI interface as the ``--int8`` option.
|
||||
|
||||
.. note::
|
||||
|
||||
8-bit weight compression is enabled by default for models larger than 1 billion parameters.
|
||||
|
||||
`Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__ also provides 4-bit weight
|
||||
compression with ``OVWeightQuantizationConfig`` class to control weight quantization parameters.
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from optimum.intel import OVModelForCausalLM, OVWeightQuantizationConfig
|
||||
import nncf
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
export=True,
|
||||
quantization_config=OVWeightQuantizationConfig(bits=4, asym=True, ratio=0.8, dataset="ptb"),
|
||||
)
|
||||
|
||||
|
||||
The optimized model can be saved as usual with a call to ``save_pretrained()``.
|
||||
For more details on compression options, refer to the :doc:`weight compression guide <weight_compression>`.
|
||||
|
||||
.. note::
|
||||
|
||||
OpenVINO also supports 4-bit models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized
|
||||
with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it.
|
||||
|
||||
Below are some examples of using Optimum-Intel for model conversion and inference:
|
||||
|
||||
* `Instruction following using Databricks Dolly 2.0 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/240-dolly-2-instruction-following/240-dolly-2-instruction-following.ipynb>`__
|
||||
* `Create an LLM-powered Chatbot using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-llm-chatbot.ipynb>`__
|
||||
|
||||
.. note::
|
||||
|
||||
Optimum-Intel can be used for other generative AI models. See `Stable Diffusion v2.1 using Optimum-Intel OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/236-stable-diffusion-v2/236-stable-diffusion-v2-optimum-demo.ipynb>`__ and `Image generation with Stable Diffusion XL and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.ipynb>`__ for more examples.
|
||||
|
||||
Inference Example
|
||||
############################################################
|
||||
|
||||
For Hugging Face models, the ``AutoTokenizer`` and the ``pipeline`` function are used to create
|
||||
an inference pipeline. This setup allows for easy text processing and model interaction:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from optimum.intel import OVModelForCausalLM
|
||||
# new imports for inference
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
# load the model
|
||||
model_id = "meta-llama/Llama-2-7b-chat-hf"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||||
|
||||
# inference
|
||||
prompt = "The weather is:"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(**inputs, max_new_tokens=50)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
|
||||
.. note::
|
||||
|
||||
Converting LLMs on the fly every time to OpenVINO IR is a resource intensive task.
|
||||
It is a good practice to convert the model once, save it in a folder and load it for inference.
|
||||
|
||||
By default, inference will run on CPU. To switch to a different device, the ``device`` attribute
|
||||
from the ``from_pretrained`` function can be used. The device naming convention is the
|
||||
same as in OpenVINO native API:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True, device="GPU")
|
||||
|
||||
Enabling OpenVINO Runtime Optimizations
|
||||
############################################################
|
||||
|
||||
OpenVINO runtime provides a set of optimizations for more efficient LLM inference. This includes **Dynamic quantization** of activations of 4/8-bit quantized MatMuls and **KV-cache quantization**.
|
||||
|
||||
* **Dynamic quantization** enables quantization of activations of MatMul operations that have 4 or 8-bit quantized weights (see :doc:`LLM Weight Compression <weight_compression>`).
|
||||
It improves inference latency and throughput of LLMs, though it may cause insignificant deviation in generation accuracy. Quantization is performed in a
|
||||
group-wise manner, with configurable group size. It means that values in a group share quantization parameters. Larger group sizes lead to faster inference but lower accuracy. Recommended group size values are: ``32``, ``64``, or ``128``. To enable Dynamic quantization, use the corresponding
|
||||
inference property as follows:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
ov_config={"DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY"}
|
||||
)
|
||||
|
||||
|
||||
|
||||
* **KV-cache quantization** allows lowering the precision of Key and Value cache in LLMs. This helps reduce memory consumption during inference, improving latency and throughput. KV-cache can be quantized into the following precisions:
|
||||
``u8``, ``bf16``, ``f16``. If ``u8`` is used, KV-cache quantization is also applied in a group-wise manner. Thus, it can use ``DYNAMIC_QUANTIZATION_GROUP_SIZE`` value if defined.
|
||||
Otherwise, the group size ``32`` is used by default. KV-cache quantization can be enabled as follows:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
ov_config={"KV_CACHE_PRECISION": "u8", "DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY"}
|
||||
)
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
Currently, both Dynamic quantization and KV-cache quantization are available for CPU device.
|
||||
|
||||
|
||||
Working with Models Tuned with LoRA
|
||||
#########################################
|
||||
|
||||
Low-rank Adaptation (LoRA) is a popular method to tune Generative AI models to a downstream task
|
||||
or custom data. However, it requires some extra steps to be done for efficient deployment using
|
||||
the Hugging Face API. Namely, the trained adapters should be fused into the baseline model to
|
||||
avoid extra computation. This is how it can be done for LLMs:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model_id = "meta-llama/Llama-2-7b-chat-hf"
|
||||
lora_adaptor = "./lora_adaptor"
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, use_cache=True)
|
||||
model = PeftModelForCausalLM.from_pretrained(model, lora_adaptor)
|
||||
model.merge_and_unload()
|
||||
model.get_base_model().save_pretrained("fused_lora_model")
|
||||
|
||||
|
||||
Now the model can be converted to OpenVINO using Optimum Intel Python API or CLI interfaces mentioned above.
|
||||
|
||||
|
||||
Additional Resources
|
||||
#####################
|
||||
|
||||
* `Optimum Intel documentation <https://huggingface.co/docs/optimum/intel/inference>`__
|
||||
* :doc:`LLM Weight Compression <weight_compression>`
|
||||
* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__
|
||||
* `Hugging Face Transformers <https://huggingface.co/docs/transformers/index>`__
|
||||
* `Generation with LLMs <https://huggingface.co/docs/transformers/llm_tutorial>`__
|
||||
* `Pipeline class <https://huggingface.co/docs/transformers/main_classes/pipelines>`__
|
||||
* `GenAI Pipeline Repository <https://github.com/openvinotoolkit/openvino.genai>`__
|
||||
* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_contrib/tree/master/modules/custom_operations/user_ie_extensions/tokenizer/python>`__
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
.. {#llm_inference_native_ov}
|
||||
|
||||
Inference with Native OpenVINO
|
||||
===============================
|
||||
|
||||
To run Generative AI models using native OpenVINO APIs you need to follow regular **Сonvert -> Optimize -> Deploy** path with a few simplifications.
|
||||
|
||||
To convert model from Hugging Face you can use Optimum-Intel export feature that allows to export model in OpenVINO format without invoking conversion API and tools directly, as it is shown above.
|
||||
In this case, the conversion process is a bit more simplified. You can still use a regular conversion path if model comes from outside of Hugging Face ecosystem, i.e., in source framework format (PyTorch, etc.)
|
||||
|
||||
Model optimization can be performed within Hugging Face or directly using NNCF as described in the :doc:`weight compression guide <weight_compression>`.
|
||||
|
||||
.. note::
|
||||
|
||||
It is recommended to use models in 4-bit precision, as maintaining the model in its original precision may result in significantly decreased performance.
|
||||
|
||||
Inference code that uses native API cannot benefit from Hugging Face pipelines. You need to write your custom code or take it from the available examples. Below are some examples of popular Generative AI scenarios:
|
||||
|
||||
* In case of LLMs for text generation, you need to handle tokenization, inference and token selection loop, and de-tokenization. If token selection involves beam search, it also needs to be written.
|
||||
* For image generation models, you need to make a pipeline that includes several model inferences: inference for source (e.g., text) encoder models, inference loop for diffusion process and inference for decoding part. Scheduler code is also required.
|
||||
|
||||
To write such pipelines, you can follow the examples provided as part of OpenVINO:
|
||||
|
||||
* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
|
||||
* `OpenVINO Latent Consistency Model C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/lcm_dreamshaper_v7/cpp>`__
|
||||
* `OpenVINO Stable Diffusion (with LoRA) C++ image generation pipeline <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/stable_diffusion_1_5/cpp>`__
|
||||
|
||||
To perform inference, models must be first converted to OpenVINO IR format using Hugging Face Optimum-Intel API.
|
||||
|
||||
An inference pipeline for a text generation LLM is set up in the following stages:
|
||||
|
||||
1. Read and compile the model in OpenVINO IR.
|
||||
2. Pre-process text prompt with a tokenizer and set the result as model inputs.
|
||||
3. Run token generation loop.
|
||||
4. De-tokenize outputs.
|
||||
|
||||
Prerequisites
|
||||
########################
|
||||
|
||||
Linux operating system (as of the current version).
|
||||
|
||||
**Installation**
|
||||
|
||||
1. Create a virtual environment
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
python -m venv openvino_llm
|
||||
|
||||
``openvino_llm`` is an example name; you can choose any name for your environment.
|
||||
|
||||
2. Activate the virtual environment
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
source openvino_llm/bin/activate
|
||||
|
||||
3. Install OpenVINO tokenizers and dependencies
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pip install transformers optimum[transformers]
|
||||
|
||||
|
||||
Convert Hugging Face tokenizer and model to OpenVINO IR format
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
**Convert Tokenizer**
|
||||
|
||||
`OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_contrib/tree/master/modules/custom_operations/user_ie_extensions/tokenizer/python#openvino-tokenizers>`__
|
||||
come equipped with a CLI tool that facilitates the conversion of tokenizers
|
||||
from either the Hugging Face Hub or those saved locally to the OpenVINO IR format:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
convert_tokenizer microsoft/Llama2-7b-WhoIsHarryPotter --with-detokenizer -o openvino_tokenizer
|
||||
|
||||
In this example, the ``microsoft/Llama2-7b-WhoIsHarryPotter tokenizer`` is transformed from the Hugging
|
||||
Face hub. You can substitute this tokenizer with one of your preference. You can also rename
|
||||
the output directory (``openvino_tokenizer``).
|
||||
|
||||
**Convert Model**
|
||||
|
||||
The optimum-cli command can be used for converting a Hugging Face model to the OpenVINO IR model format.
|
||||
Learn more in Loading an LLM with OpenVINO.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
optimum-cli export openvino --convert-tokenizer --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 openvino_model
|
||||
|
||||
Full OpenVINO Text Generation Pipeline
|
||||
######################################################################
|
||||
|
||||
1. Import and Compile Models
|
||||
+++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
Use the model and tokenizer converted from the previous step:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import numpy as np
|
||||
from openvino import compile_model
|
||||
|
||||
# Compile the tokenizer, model, and detokenizer using OpenVINO. These files are XML representations of the models optimized for OpenVINO
|
||||
compiled_tokenizer = compile_model("openvino_tokenizer.xml")
|
||||
compiled_model = compile_model("openvino_model.xml")
|
||||
compiled_detokenizer = compile_model("openvino_detokenizer.xml")
|
||||
|
||||
2. Tokenize and Transform Input
|
||||
+++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
Tokenization is a mandatory step in the process of generating text using LLMs. Tokenization
|
||||
converts the input text into a sequence of tokens, which are essentially the format that the
|
||||
model can understand and process. The input text string must be tokenized and set up in the
|
||||
structure expected by the model before running inference.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
text_input = ["Quick brown fox was"]
|
||||
ov_input = compiled_tokenizer(text_input)
|
||||
|
||||
3. Generate Tokens
|
||||
+++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
The core of text generation lies in the inference and token selection loop. In each iteration
|
||||
of this loop, the model runs inference on the input sequence, generates and selects a new token,
|
||||
and appends it to the existing sequence.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Define the number of new tokens to generate
|
||||
new_tokens_size = 10
|
||||
|
||||
# Determine the size of the existing prompt
|
||||
prompt_size = ov_input["input_ids"].shape[-1]
|
||||
|
||||
# Prepare the input dictionary for the model
|
||||
# It combines existing tokens with additional space for new tokens
|
||||
input_dict = {
|
||||
output.any_name: np.hstack([tensor, np.zeros(shape=(1, new_tokens_size), dtype=np.int_)])
|
||||
for output, tensor in ov_input.items()
|
||||
}
|
||||
|
||||
# Generate new tokens iteratively
|
||||
for idx in range(prompt_size, prompt_size + new_tokens_size):
|
||||
# Get output from the model
|
||||
output = compiled_model(input_dict)["token_ids"]
|
||||
# Update the input_ids with newly generated token
|
||||
input_dict["input_ids"][:, idx] = output[:, idx - 1]
|
||||
# Update the attention mask to include the new token
|
||||
input_dict["attention_mask"][:, idx] = 1
|
||||
|
||||
4. Decode and Display Output
|
||||
+++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
The final step in the process is de-tokenization, where the sequence of token IDs generated by
|
||||
the model is converted back into human-readable text.
|
||||
This step is essential for interpreting the model's output.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Extract token IDs for the final output
|
||||
ov_token_ids = input_dict["input_ids"]
|
||||
# Decode the model output back to string
|
||||
ov_output = compiled_detokenizer(ov_token_ids)["string_output"]
|
||||
print(f"OpenVINO output string: `{ov_output}`")
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Example output:
|
||||
['<s> Quick brown fox was walking through the forest. He was looking for something']
|
||||
|
||||
|
||||
Additional Resources
|
||||
####################
|
||||
|
||||
* `Text generation C++ samples that support most popular models like LLaMA 2 <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__
|
||||
* `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__
|
||||
* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_contrib/tree/master/modules/custom_operations/user_ie_extensions/tokenizer/python#openvino-tokenizers>`__
|
||||
* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__
|
||||
* :doc:`Stateful Models Low-Level Details <openvino_docs_OV_UG_stateful_models_intro>`
|
||||
* :doc:`Working with Textual Data <openvino_docs_OV_UG_string_tensors>`
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
.. {#native_vs_hugging_face_api}
|
||||
|
||||
Large Language Models Inference Guide
|
||||
========================================
|
||||
|
||||
.. meta::
|
||||
:description: Explore learning materials, including interactive
|
||||
Python tutorials and sample console applications that explain
|
||||
how to use OpenVINO features.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
LLM Inference with Optimum Intel <llm_inference>
|
||||
LLM Inference with OpenVINO API <llm_inference_native_ov>
|
||||
|
||||
|
||||
Large Language Models (LLMs) like GPT are transformative deep learning networks
|
||||
capable of a broad range of natural language tasks, from text generation to language translation.
|
||||
OpenVINO optimizes the deployment of these models, enhancing their performance and integration into various applications.
|
||||
This guide shows how to use LLMs with OpenVINO, from model loading and conversion to advanced use cases.
|
||||
|
||||
The advantages of using OpenVINO for LLM deployment:
|
||||
|
||||
● **OpenVINO offers optimized LLM inference**; provides a full C/C++ API, leading to faster operation
|
||||
than Python-based runtimes; includes a Python API for rapid development,
|
||||
with the option for further optimization in C++.
|
||||
|
||||
● **Compatible with diverse hardware**, supports CPUs, GPUs, and neural accelerators
|
||||
across ARM and x86/x64 architectures, integrated Intel® Processor Graphics,
|
||||
discrete Intel® Arc™ A-Series Graphics, and discrete Intel® Data Center GPU Flex Series;
|
||||
features automated optimization to maximize performance on target hardware.
|
||||
|
||||
● **Requires fewer dependencies** than frameworks like
|
||||
Hugging Face and PyTorch, resulting in a smaller binary size and reduced
|
||||
memory footprint, making deployments easier and updates more manageable.
|
||||
|
||||
● **Provides compression and precision management techniques** such as 8-bit and 4-bit weight compression,
|
||||
including embedding layers, and storage format reduction.
|
||||
This includes fp16 precision for non-compressed models and int8/int4 for compressed models, like GPTQ models from Hugging Face.
|
||||
|
||||
● **Supports a wide range of deep learning models and architectures** including text, image,
|
||||
and audio generative models like Llama 2, MPT, OPT, Stable Diffusion, Stable Diffusion XL.
|
||||
This enables the development of multimodal applications, allowing for write-once, deploy-anywhere capabilities.
|
||||
|
||||
● **Enhances inference capabilities**: fused inference primitives such as Scaled
|
||||
Dot Product Attention, Rotary Positional Embedding, Group Query Attention, and Mixture of Experts.
|
||||
It also offers advanced features like in-place KV-cache, dynamic quantization,
|
||||
KV-cache quantization and encapsulation, dynamic beam size configuration, and speculative
|
||||
sampling.
|
||||
|
||||
● **Provides stateful model optimization**: models from the Hugging Face Transformers are converted
|
||||
into a stateful form, optimizing inference performance and memory usage in long running text generation
|
||||
tasks by managing past KV-cache tensors more efficiently internally. This feature is automatically activated
|
||||
for many supported models, while unsupported ones remain stateless.
|
||||
Learn more about the :doc:`Stateful models and State API <openvino_docs_OV_UG_stateful_models_intro>`.
|
||||
|
||||
OpenVINO offers two main paths for Generative AI use cases:
|
||||
|
||||
* **Hugging Face**: use OpenVINO as a backend for Hugging Face frameworks (transformers, diffusers) through the `Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__ extension.
|
||||
|
||||
* **Native OpenVINO**: use OpenVINO native APIs (Python and C++) with `custom pipeline code <https://github.com/openvinotoolkit/openvino.genai>`__.
|
||||
|
||||
In both cases, the OpenVINO runtime is used for inference,
|
||||
and OpenVINO tools are used for optimization. The main differences are
|
||||
in footprint size, ease of use and customizability.
|
||||
|
||||
The Hugging Face API is easy to learn, provides a simple interface and hides
|
||||
the complexity of model initialization and text generation for a better developer experience.
|
||||
However, it has more dependencies, less customization, and cannot be ported to C/C++.
|
||||
|
||||
The Native OpenVINO API requires fewer dependencies,
|
||||
mininmizing the application footprint, and enables the use
|
||||
of generative models in C++ applications. However, it requires explicit
|
||||
implementation of the text generation loop, tokenization functions,
|
||||
and scheduler functions used in a typical LLM pipeline.
|
||||
|
||||
It is recommended to start with Hugging Face frameworks to experiment with
|
||||
different models and scenarios. Then the model can be used with OpenVINO native APIs
|
||||
if it needs to be optimized further. Optimum Intel provides interfaces that enable model optimization (weight compression)
|
||||
using `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__,
|
||||
and export models to the OpenVINO model format for use in native API applications.
|
||||
|
||||
* To proceed with **Hugging Face API**, read :doc:`LLM Inference with Hugging Face and Optimum Intel <llm_inference>` guide.
|
||||
|
||||
* To proceed with **Native OpenVINO API**, go to :doc:`LLM Inference with Native OpenVINO <llm_inference_native_ov>` page.
|
||||
|
||||
The table below summarizes the differences between Hugging Face and the native OpenVINO API approaches.
|
||||
|
||||
.. list-table::
|
||||
:widths: 20 25 55
|
||||
:header-rows: 1
|
||||
|
||||
* -
|
||||
- Hugging Face through OpenVINO
|
||||
- OpenVINO Native API
|
||||
* - Model support
|
||||
- Supports transformer-based models such as LLMs
|
||||
- Supports all model architectures from most frameworks
|
||||
* - APIs
|
||||
- Python (Hugging Face API)
|
||||
- Python, C++ (OpenVINO API)
|
||||
* - Model Format
|
||||
- Source Framework / OpenVINO
|
||||
- Source Framework / OpenVINO
|
||||
* - Inference code
|
||||
- Hugging Face based
|
||||
- Custom inference pipelines
|
||||
* - Additional dependencies
|
||||
- Many Hugging Face dependencies
|
||||
- Lightweight (e.g. numpy, etc.)
|
||||
* - Application footprint
|
||||
- Large
|
||||
- Small
|
||||
* - Pre/post-processing and glue code
|
||||
- Provided through high-level Hugging Face APIs
|
||||
- Must be custom implemented (see OpenVINO samples and notebooks)
|
||||
* - Performance
|
||||
- Good, but less efficient compared to native APIs
|
||||
- Inherent speed advantage with C++, but requires hands-on optimization
|
||||
* - Flexibility
|
||||
- Constrained to Hugging Face API
|
||||
- High flexibility with Python and C++; allows custom coding
|
||||
* - Learning Curve and Effort
|
||||
- Lower learning curve; quick to integrate
|
||||
- Higher learning curve; requires more effort in integration
|
||||
* - Ideal Use Case
|
||||
- Ideal for quick prototyping and Python-centric projects
|
||||
- Best suited for high-performance, resource-optimized production environments
|
||||
* - Model Serving
|
||||
- Paid service, based on CPU/GPU usage with Hugging Face
|
||||
- Free code solution, run script for own server; costs may incur for cloud services like AWS but generally cheaper than Hugging Face rates
|
||||
|
||||
|
||||
|
|
@ -1,285 +0,0 @@
|
|||
.. {#gen_ai_guide}
|
||||
|
||||
Optimize and Deploy Generative AI Models
|
||||
========================================
|
||||
|
||||
|
||||
Generative AI is an innovative technique that creates new data, such as text, images, video,
|
||||
or audio, using neural networks. OpenVINO accelerates Generative AI use cases as they mostly
|
||||
rely on model inference, allowing for faster development and better performance. When it
|
||||
comes to generative models, OpenVINO supports:
|
||||
|
||||
* Conversion, optimization and inference for text, image and audio generative models, for
|
||||
example, Llama 2, MPT, OPT, Stable Diffusion, Stable Diffusion XL, etc.
|
||||
* 8-bit and 4-bit weight compression including compression of Embedding layers.
|
||||
* Storage format reduction (fp16 precision for non-compressed models and int8/int4 for compressed
|
||||
models), including GPTQ models from Hugging Face.
|
||||
* Inference on CPU and GPU platforms, including integrated Intel® Processor Graphics,
|
||||
discrete Intel® Arc™ A-Series Graphics, and discrete Intel® Data Center GPU Flex Series.
|
||||
* Fused inference primitives, for example, Scaled Dot Product Attention, Rotary Positional Embedding,
|
||||
Group Query Attention, Mixture of Experts, etc.
|
||||
* In-place KV-cache, Dynamic quantization, KV-cache quantization and encapsulation.
|
||||
* Dynamic beam size configuration, Speculative sampling.
|
||||
|
||||
|
||||
OpenVINO offers two main paths for Generative AI use cases:
|
||||
|
||||
* Using OpenVINO as a backend for Hugging Face frameworks (transformers, diffusers) through
|
||||
the `Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__ extension.
|
||||
* Using OpenVINO native APIs (Python and C++) with `custom pipeline code <https://github.com/openvinotoolkit/openvino.genai>`__.
|
||||
|
||||
|
||||
In both cases, OpenVINO runtime and tools are used, the difference is mostly in the preferred
|
||||
API and the final solution's footprint. Native APIs enable the use of generative models in
|
||||
C++ applications, ensure minimal runtime dependencies, and minimize application footprint.
|
||||
The Native APIs approach requires the implementation of glue code (generation loop, text
|
||||
tokenization, or scheduler functions), which is hidden within Hugging Face libraries for a
|
||||
better developer experience.
|
||||
|
||||
It is recommended to start with Hugging Face frameworks. Experiment with different models and
|
||||
scenarios to find your fit, and then consider converting to OpenVINO native APIs based on your
|
||||
specific requirements.
|
||||
|
||||
Optimum Intel provides interfaces that enable model optimization (weight compression) using
|
||||
`Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__,
|
||||
and export models to the OpenVINO model format for use in native API applications.
|
||||
|
||||
The table below summarizes the differences between Hugging Face and Native APIs approaches.
|
||||
|
||||
.. list-table::
|
||||
:widths: 20 25 55
|
||||
:header-rows: 1
|
||||
|
||||
* -
|
||||
- Hugging Face through OpenVINO
|
||||
- OpenVINO Native API
|
||||
* - Model support
|
||||
- Broad set of Models
|
||||
- Broad set of Models
|
||||
* - APIs
|
||||
- Python (Hugging Face API)
|
||||
- Python, C++ (OpenVINO API)
|
||||
* - Model Format
|
||||
- Source Framework / OpenVINO
|
||||
- OpenVINO
|
||||
* - Inference code
|
||||
- Hugging Face based
|
||||
- Custom inference pipelines
|
||||
* - Additional dependencies
|
||||
- Many Hugging Face dependencies
|
||||
- Lightweight (e.g. numpy, etc.)
|
||||
* - Application footprint
|
||||
- Large
|
||||
- Small
|
||||
* - Pre/post-processing and glue code
|
||||
- Available at Hugging Face out-of-the-box
|
||||
- OpenVINO samples and notebooks
|
||||
* - Performance
|
||||
- Good
|
||||
- Best
|
||||
|
||||
|
||||
Running Generative AI Models using Hugging Face Optimum Intel
|
||||
##############################################################
|
||||
|
||||
Prerequisites
|
||||
+++++++++++++++++++++++++++
|
||||
|
||||
* Create a Python environment.
|
||||
* Install Optimum Intel:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
pip install optimum[openvino,nncf]
|
||||
|
||||
|
||||
To start using OpenVINO as a backend for Hugging Face, change the original Hugging Face code in two places:
|
||||
|
||||
.. code-block:: diff
|
||||
|
||||
-from transformers import AutoModelForCausalLM
|
||||
+from optimum.intel import OVModelForCausalLM
|
||||
|
||||
model_id = "meta-llama/Llama-2-7b-chat-hf"
|
||||
-model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
+model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||||
|
||||
|
||||
After that, you can call ``save_pretrained()`` method to save model to the folder in the OpenVINO
|
||||
Intermediate Representation and use it further.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model.save_pretrained(model_dir)
|
||||
|
||||
|
||||
Alternatively, you can download and convert the model using CLI interface:
|
||||
``optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf llama_openvino``.
|
||||
In this case, you can load the converted model in OpenVINO representation directly from the disk:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model_id = "llama_openvino"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id)
|
||||
|
||||
|
||||
By default, inference will run on CPU. To select a different inference device, for example, GPU,
|
||||
add ``device="GPU"`` to the ``from_pretrained()`` call. To switch to a different device after
|
||||
the model has been loaded, use the ``.to()`` method. The device naming convention is the same
|
||||
as in OpenVINO native API:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model.to("GPU")
|
||||
|
||||
|
||||
Optimum-Intel API also provides out-of-the-box model optimization through weight compression
|
||||
using NNCF which substantially reduces the model footprint and inference latency:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True, load_in_8bit=True)
|
||||
|
||||
|
||||
Weight compression is applied by default to models larger than one billion parameters and is
|
||||
also available for CLI interface as the ``--int8`` option.
|
||||
|
||||
.. note::
|
||||
|
||||
8-bit weight compression is enabled by default for models larger than 1 billion parameters.
|
||||
|
||||
`Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>`__ also provides 4-bit weight compression with ``OVWeightQuantizationConfig`` class to control weight quantization parameters.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from optimum.intel import OVModelForCausalLM, OVWeightQuantizationConfig
|
||||
import nncf
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
export=True,
|
||||
quantization_config=OVWeightQuantizationConfig(bits=4, asym=True, ratio=0.8, dataset="ptb"),
|
||||
)
|
||||
|
||||
|
||||
The optimized model can be saved as usual with a call to ``save_pretrained()``.
|
||||
For more details on compression options, refer to the :doc:`weight compression guide <weight_compression>`.
|
||||
|
||||
.. note::
|
||||
|
||||
OpenVINO also supports 4-bit models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized
|
||||
with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it.
|
||||
|
||||
Another optimization that is applied by default when using ``OVModelForCausalLM`` class is transformation of the model to a stateful form.
|
||||
This transformation further improves inference performance and decreases amount of allocated runtime memory in long running text generation scenarious.
|
||||
It is achieved by hiding inputs and outputs of the model that represent past KV-cache tensors, and handling them inside the model in a more efficient way.
|
||||
This feature is activated automatically for a wide range of supported text generation models, keeping not supported models in a regular, stateless form.
|
||||
|
||||
Model usage are identical for stateful and stateless models as long as Optimum-Intel API is used because KV-cache handling is an internal detail of the text-generation API of Transformers library.
|
||||
But a form of a model matterns in case when exported from Optimum-Intel OpenVINO model IR is used in an application implemented with native OpenVINO API, because stateful and stateless models have different number of inputs and outputs.
|
||||
Please refer to a dedicated section of this document below for more information about using native OpenVINO API.
|
||||
|
||||
Below are some examples of using Optimum-Intel for model conversion and inference:
|
||||
|
||||
* `Stable Diffusion v2.1 using Optimum-Intel OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/236-stable-diffusion-v2/236-stable-diffusion-v2-optimum-demo.ipynb>`__
|
||||
* `Image generation with Stable Diffusion XL and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/248-stable-diffusion-xl/248-stable-diffusion-xl.ipynb>`__
|
||||
* `Instruction following using Databricks Dolly 2.0 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/240-dolly-2-instruction-following/240-dolly-2-instruction-following.ipynb>`__
|
||||
* `Create an LLM-powered Chatbot using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-llm-chatbot.ipynb>`__
|
||||
|
||||
Stateful Model Optimization
|
||||
+++++++++++++++++++++++++++
|
||||
|
||||
When you use the ``OVModelForCausalLM`` class, the model is transformed into a stateful form by default for optimization.
|
||||
This transformation improves inference performance and decreases runtime memory usage in long running text generation tasks.
|
||||
It is achieved by hiding the model's inputs and outputs that represent past KV-cache tensors, and handling them inside the model in a more efficient way.
|
||||
This feature is activated automatically for many supported text generation models, while unsupported models remain in a regular, stateless form.
|
||||
|
||||
Model usage remains the same for stateful and stateless models with the Optimum-Intel API, as KV-cache is handled internally by text-generation API of Transformers library.
|
||||
The model's form matters when an OpenVINO IR model is exported from Optimum-Intel and used in an application with the native OpenVINO API.
|
||||
This is because stateful and stateless models have a different number of inputs and outputs.
|
||||
Learn more about the `native OpenVINO API <Running-Generative-AI-Models-using-Native-OpenVINO-APIs>`__.
|
||||
|
||||
Enabling OpenVINO Runtime Optimizations
|
||||
+++++++++++++++++++++++++++++++++++++++
|
||||
OpenVINO runtime provides a set of optimizations for more efficient LLM inference. This includes **Dynamic quantization** of activations of 4/8-bit quantized MatMuls and **KV-cache quantization**.
|
||||
|
||||
* **Dynamic quantization** enables quantization of activations of MatMul operations that have 4 or 8-bit quantized weights (see :doc:`LLM Weight Compression <weight_compression>`).
|
||||
It improves inference latency and throughput of LLMs, though it may cause insignificant deviation in generation accuracy. Quantization is performed in a
|
||||
group-wise manner, with configurable group size. It means that values in a group share quantization parameters. Larger group sizes lead to faster inference but lower accuracy. Recommended group size values are: ``32``, ``64``, or ``128``. To enable Dynamic quantization, use the corresponding
|
||||
inference property as follows:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
ov_config={"DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY"}
|
||||
)
|
||||
|
||||
|
||||
|
||||
* **KV-cache quantization** allows lowering the precision of Key and Value cache in LLMs. This helps reduce memory consumption during inference, improving latency and throughput. KV-cache can be quantized into the following precisions:
|
||||
``u8``, ``bf16``, ``f16``. If ``u8`` is used, KV-cache quantization is also applied in a group-wise manner. Thus, it can use ``DYNAMIC_QUANTIZATION_GROUP_SIZE`` value if defined.
|
||||
Otherwise, the group size ``32`` is used by default. KV-cache quantization can be enabled as follows:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = OVModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
ov_config={"KV_CACHE_PRECISION": "u8", "DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY"}
|
||||
)
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
Currently, both Dynamic quantization and KV-cache quantization are available for CPU device.
|
||||
|
||||
|
||||
Working with Models Tuned with LoRA
|
||||
++++++++++++++++++++++++++++++++++++
|
||||
|
||||
Low-rank Adaptation (LoRA) is a popular method to tune Generative AI models to a downstream task or custom data. However, it requires some extra steps to be done for efficient deployment using the Hugging Face API. Namely, the trained adapters should be fused into the baseline model to avoid extra computation. This is how it can be done for Large Language Models (LLMs):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model_id = "meta-llama/Llama-2-7b-chat-hf"
|
||||
lora_adaptor = "./lora_adaptor"
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, use_cache=True)
|
||||
model = PeftModelForCausalLM.from_pretrained(model, lora_adaptor)
|
||||
model.merge_and_unload()
|
||||
model.get_base_model().save_pretrained("fused_lora_model")
|
||||
|
||||
|
||||
Now the model can be converted to OpenVINO using Optimum Intel Python API or CLI interfaces mentioned above.
|
||||
|
||||
Running Generative AI Models using Native OpenVINO APIs
|
||||
########################################################
|
||||
|
||||
To run Generative AI models using native OpenVINO APIs, you need to follow regular **Convert -> Optimize -> Deploy** path with a few simplifications.
|
||||
|
||||
The recommended way for converting a Hugging Face model is to use the Optimum-Intel export feature. This feature enables model export in OpenVINO format without directly invoking conversion API and tools, as demonstrated above.
|
||||
The conversion process is significantly simplified as Optimum-Intel provides the necessary conversion parameters. These parameters are often model-specific and require knowledge of various model input properties.
|
||||
|
||||
Moreover, Optimum-Intel applies several model optimizations, such as weight compression and using stateful form by default, that further simplify the model exporting flow.
|
||||
You can still use the regular conversion path if the model comes from outside the Hugging Face ecosystem, such as in its source framework format (PyTorch, TensorFlow, etc.).
|
||||
|
||||
Model optimization can be performed within Hugging Face or directly using NNCF as described in the :doc:`weight compression guide <weight_compression>`.
|
||||
|
||||
Inference code that uses native API cannot benefit from Hugging Face pipelines. You need to write your custom code or take it from the available examples. Below are some examples of popular Generative AI scenarios:
|
||||
|
||||
* In case of LLMs for text generation, you need to handle tokenization, inference and token sampling, and de-tokenization. If token sampling involves beam search, you need to implement it as well. This is covered in details by `C++ Text Generation Samples <https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp>`__.
|
||||
* For image generation models, you need to make a pipeline that includes several model inferences: inference for source (for example, text) encoder models, inference loop for diffusion process and inference for decoding part. Scheduler code is also required. `C++ Implementation of Stable Diffusion <https://github.com/openvinotoolkit/openvino.genai/tree/master/image_generation/stable_diffusion_1_5/cpp>`__ is a good reference point.
|
||||
|
||||
|
||||
Additional Resources
|
||||
#####################
|
||||
|
||||
* `Optimum Intel documentation <https://huggingface.co/docs/optimum/intel/inference>`__
|
||||
* :doc:`LLM Weight Compression <weight_compression>`
|
||||
* `Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>`__
|
||||
* `GenAI Pipeline Repository <https://github.com/openvinotoolkit/openvino.genai>`__
|
||||
* `OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_contrib/tree/master/modules/custom_operations/user_ie_extensions/tokenizer/python>`__
|
||||
* :doc:`Stateful Models Low-Level Details <openvino_docs_OV_UG_stateful_models_intro>`
|
||||
* :doc:`Working with Textual Data <openvino_docs_OV_UG_string_tensors>`
|
||||
|
|
@ -3,21 +3,47 @@
|
|||
Weight Compression
|
||||
==================
|
||||
|
||||
Weight compression is a technique for enhancing the efficiency of models,
|
||||
especially those with large memory requirements. This method reduces the model's
|
||||
memory footprint, a crucial factor for Large Language Models (LLMs).
|
||||
|
||||
Enhancing Model Efficiency with Weight Compression
|
||||
##################################################################
|
||||
Unlike full model quantization, where weights and activations are quantized,
|
||||
weight compression in `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__
|
||||
only targets the model's weights. This approach
|
||||
allows the activations to remain as floating-point numbers, preserving most
|
||||
of the model's accuracy while improving its speed and reducing
|
||||
its size.
|
||||
|
||||
Weight compression aims to reduce the memory footprint of a model. It can also lead to significant performance improvement for large memory-bound models, such as Large Language Models (LLMs). LLMs and other models, which require extensive memory to store the weights during inference, can benefit from weight compression in the following ways:
|
||||
The reduction in size is especially noticeable with larger models,
|
||||
for instance the 7 billion parameter Llama 2 model can be reduced
|
||||
from about 25GB to 4GB using 4-bit weight compression. With smaller models (i.e. less than 1B parameters),
|
||||
weight compression may result in more accuracy reduction than with larger models.
|
||||
|
||||
- enabling the inference of exceptionally large models that cannot be accommodated in the memory of the device;
|
||||
- improving the inference performance of the models by reducing the latency of the memory access when computing the operations with weights, for example, Linear layers.
|
||||
LLMs and other models that require
|
||||
extensive memory to store the weights during inference can benefit
|
||||
from weight compression as it:
|
||||
|
||||
* enables inference of exceptionally large models that cannot be accommodated in the device memory;
|
||||
|
||||
* reduces storage and memory overhead, making models more lightweight and less resource intensive for deployment;
|
||||
|
||||
* improves inference speed by reducing the latency of memory access when computing the operations with weights, for example, Linear layers. The weights are smaller and thus faster to load from memory;
|
||||
|
||||
* unlike quantization, does not require sample data to calibrate the range of activation values.
|
||||
|
||||
Currently, `NNCF <https://github.com/openvinotoolkit/nncf>`__
|
||||
provides weight quantization to 8 and 4-bit integer data types as a compression
|
||||
method primarily designed to optimize LLMs.
|
||||
|
||||
Currently, `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__ provides weight quantization to 8 and 4-bit integer data types as a compression method primarily designed to optimize LLMs. The main difference between weights compression and full model quantization is that activations remain floating-point in the case of weight compression, resulting in better accuracy. Weight compression for LLMs provides a solid inference performance improvement which is on par with the performance of the full model quantization. In addition, weight compression is data-free and does not require a calibration dataset, making it easy to use.
|
||||
|
||||
Compress Model Weights
|
||||
######################
|
||||
|
||||
- **8-bit weight quantization** - this method is aimed at accurate optimization of the model, which usually leads to significant performance improvements for Transformer-based models. Models with 8-bit compressed weights are performant on the vast majority of supported CPU and GPU platforms.
|
||||
**8-bit weight quantization** method offers a balance between model size reduction and
|
||||
maintaining accuracy, which usually leads to significant performance improvements for
|
||||
Transformer-based models. Models with 8-bit compressed weights are performant on the
|
||||
vast majority of supported CPU and GPU platforms.
|
||||
|
||||
|
||||
The code snippet below shows how to do 8-bit quantization of the model weights represented in OpenVINO IR using NNCF:
|
||||
|
||||
|
|
@ -25,61 +51,234 @@ The code snippet below shows how to do 8-bit quantization of the model weights r
|
|||
|
||||
.. tab-item:: OpenVINO
|
||||
:sync: openvino
|
||||
|
||||
|
||||
.. doxygensnippet:: docs/optimization_guide/nncf/code/weight_compression_openvino.py
|
||||
:language: python
|
||||
:fragment: [compression_8bit]
|
||||
|
||||
Now, the model is ready for compilation and inference. It can be also saved into a compressed format, resulting in a smaller binary file.
|
||||
Now, the model is ready for compilation and inference.
|
||||
It can be also saved into a compressed format, resulting in a smaller binary file.
|
||||
|
||||
- **4-bit weight quantization** - this method stands for an INT4-INT8 mixed-precision weight quantization, where INT4 is considered as the primary precision and INT8 is the backup one. It usually results in a smaller model size and lower inference latency, although the accuracy degradation could be higher, depending on the model. The method has several parameters that can provide different performance-accuracy trade-offs after optimization:
|
||||
**4-bit weight quantization** method stands for an INT4-INT8 mixed-precision weight quantization,
|
||||
where INT4 is considered as the primary precision and INT8 is the backup one.
|
||||
It usually results in a smaller model size and lower inference latency, although the accuracy
|
||||
degradation could be higher, depending on the model.
|
||||
|
||||
* ``mode`` - there are two modes to choose from: ``INT4_SYM`` - stands for INT4 symmetric weight quantization and results in faster inference and smaller model size, and ``INT4_ASYM`` - INT4 asymmetric weight quantization with variable zero-point for more accurate results.
|
||||
The table below summarizes the benefits and trade-offs for each compression type in terms of
|
||||
memory reduction, speed gain, and accuracy loss.
|
||||
|
||||
* ``group_size`` - controls the size of the group of weights that share the same quantization parameters. Smaller model size results in a more accurate optimized model but with a larger footprint and slower inference. The following group sizes are recommended: ``128``, ``64``, ``32`` (``128`` is default value)
|
||||
.. list-table::
|
||||
:widths: 25 20 20 20
|
||||
:header-rows: 1
|
||||
|
||||
* ``ratio`` - controls the ratio between INT4 and INT8 compressed layers in the model. For example, 0.8 means that 80% of layers will be compressed to INT4, while the rest will be compressed to INT8 precision.
|
||||
* -
|
||||
- Memory Reduction
|
||||
- Latency Improvement
|
||||
- Accuracy Loss
|
||||
* - INT8
|
||||
- Low
|
||||
- Medium
|
||||
- Low
|
||||
* - INT4 Symmetric
|
||||
- High
|
||||
- High
|
||||
- High
|
||||
* - INT4 Asymmetric
|
||||
- High
|
||||
- Medium
|
||||
- Medium
|
||||
|
||||
* ``dataset`` - calibration dataset for data-aware weight compression. It is required for some compression options, for example, some types ``sensitivity_metric`` can use data for precision selection.
|
||||
The INT4 method has several parameters that can provide different performance-accuracy trade-offs after optimization:
|
||||
|
||||
* ``sensitivity_metric`` - controls the metric to estimate the sensitivity of compressing layers in the bit-width selection algorithm. Some of the metrics require dataset to be provided. The following types are supported:
|
||||
* ``mode`` - there are two optimization modes: symmetric and asymmetric.
|
||||
|
||||
* ``nncf.SensitivityMetric.WEIGHT_QUANTIZATION_ERROR`` - data-free metric computed as the inverted 8-bit quantization noise. Weights with highest value of this metric can be accurately quantized channel-wise to 8-bit. The idea is to leave these weights in 8 bit, and quantize the rest of layers to 4-bit group-wise. Since group-wise is more accurate than per-channel, accuracy should not degrade.
|
||||
**Symmetric Compression** - ``INT4_SYM``
|
||||
|
||||
* ``nncf.SensitivityMetric.HESSIAN_INPUT_ACTIVATION`` - requires dataset. The average Hessian trace of weights with respect to the layer-wise quantization error multiplied by L2 norm of 8-bit quantization noise.
|
||||
INT4 Symmetric mode involves quantizing weights to an unsigned 4-bit integer symmetrically with a fixed zero point of 8. This mode is faster than the INT8, making it ideal for situations where **speed and size reduction are prioritized over accuracy**.
|
||||
|
||||
* ``nncf.SensitivityMetric.MEAN_ACTIVATION_VARIANCE`` - requires dataset. The mean variance of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
.. code-block:: python
|
||||
|
||||
* ``nncf.SensitivityMetric.MAX_ACTIVATION_VARIANCE`` - requires dataset. The maximum variance of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
from nncf import compress_weights
|
||||
from nncf import CompressWeightsMode
|
||||
|
||||
* ``nncf.SensitivityMetric.MEAN_ACTIVATION_MAGNITUDE`` - requires dataset. The mean magnitude of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
compressed_model = compress_weights(model, mode=CompressWeightsMode.INT4_SYM)
|
||||
|
||||
* ``all_layers`` - boolean parameter that enables INT4 weight quantization of all Fully-Connected and Embedding layers, including the first and last layers in the model.
|
||||
**Asymmetric Compression** - ``INT4_ASYM``
|
||||
|
||||
* ``awq`` - boolean parameter that enables the AWQ method for more accurate INT4 weight quantization. Especially helpful when the weights of all the layers are quantized to 4 bits. The method can sometimes result in reduced accuracy when used with Dynamic Quantization of activations. Requires dataset.
|
||||
INT4 Asymmetric mode also uses an unsigned 4-bit integer but quantizes weights asymmetrically with a non-fixed zero point. This mode slightly compromises speed in favor of better accuracy compared to the symmetric mode. This mode is useful when **minimal accuracy loss is crucial**, but a faster performance than INT8 is still desired.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
The example below shows data-free 4-bit weight quantization applied on top of OpenVINO IR:
|
||||
from nncf import compress_weights
|
||||
from nncf import CompressWeightsMode
|
||||
|
||||
.. tab-set::
|
||||
compressed_model = compress_weights(model, mode=CompressWeightsMode.INT4_ASYM)
|
||||
|
||||
.. tab-item:: OpenVINO
|
||||
:sync: openvino
|
||||
|
||||
.. doxygensnippet:: docs/optimization_guide/nncf/code/weight_compression_openvino.py
|
||||
:language: python
|
||||
:fragment: [compression_4bit]
|
||||
* ``group_size`` controls the size of the group of weights that share the same quantization parameters. Shared quantization parameters help to speed up the calculation of activation values as they are dequantized and quantized between layers. However, they can reduce accuracy. The following group sizes are recommended: ``128``, ``64``, ``32`` (``128`` is default value).
|
||||
|
||||
`Smaller Group Size`: Leads to a more accurate model but increases the model's footprint and reduces inference speed.
|
||||
|
||||
`Larger Group Size`: Results in faster inference and a smaller model, but might compromise accuracy.
|
||||
|
||||
* ``ratio`` controls the ratio between INT4 and INT8 compressed layers in the model. Ratio is a decimal between 0 and 1. For example, 0.8 means that 80% of layers will be compressed to INT4, while the rest will be compressed to INT8 precision. The default value for ratio is 1.
|
||||
|
||||
`Higher Ratio (more INT4)`: Reduces the model size and increase inference speed but might lead to higher accuracy degradation.
|
||||
|
||||
`Lower Ratio (more INT8)`: Maintains better accuracy but results in a larger model size and potentially slower inference.
|
||||
|
||||
In this example, 90% of the model's layers are quantized to INT4 asymmetrically with a group size of 64:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from nncf import compress_weights, CompressWeightsMode
|
||||
|
||||
# Example: Compressing weights with INT4_ASYM mode, group size of 64, and 90% INT4 ratio
|
||||
compressed_model = compress_weights(
|
||||
model,
|
||||
mode=CompressWeightsMode.INT4_ASYM,
|
||||
group_size=64,
|
||||
ratio=0.9,
|
||||
)
|
||||
|
||||
* ``dataset`` - calibration dataset for data-aware weight compression. It is required for some compression options, for example, some types ``sensitivity_metric`` can use data for precision selection.
|
||||
|
||||
* ``sensitivity_metric`` - controls the metric to estimate the sensitivity of compressing layers in the bit-width selection algorithm. Some of the metrics require dataset to be provided. The following types are supported:
|
||||
|
||||
* ``nncf.SensitivityMetric.WEIGHT_QUANTIZATION_ERROR`` - data-free metric computed as the inverted 8-bit quantization noise. Weights with highest value of this metric can be accurately quantized channel-wise to 8-bit. The idea is to leave these weights in 8 bit, and quantize the rest of layers to 4-bit group-wise. Since group-wise is more accurate than per-channel, accuracy should not degrade.
|
||||
|
||||
* ``nncf.SensitivityMetric.HESSIAN_INPUT_ACTIVATION`` - requires dataset. The average Hessian trace of weights with respect to the layer-wise quantization error multiplied by L2 norm of 8-bit quantization noise.
|
||||
|
||||
* ``nncf.SensitivityMetric.MEAN_ACTIVATION_VARIANCE`` - requires dataset. The mean variance of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
|
||||
* ``nncf.SensitivityMetric.MAX_ACTIVATION_VARIANCE`` - requires dataset. The maximum variance of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
|
||||
* ``nncf.SensitivityMetric.MEAN_ACTIVATION_MAGNITUDE`` - requires dataset. The mean magnitude of the layers' inputs multiplied by inverted 8-bit quantization noise.
|
||||
|
||||
* ``all_layers`` - boolean parameter that enables INT4 weight quantization of all Fully-Connected and Embedding layers, including the first and last layers in the model.
|
||||
|
||||
* ``awq`` - boolean parameter that enables the AWQ method for more accurate INT4 weight quantization. Especially helpful when the weights of all the layers are quantized to 4 bits. The method can sometimes result in reduced accuracy when used with Dynamic Quantization of activations. Requires dataset.
|
||||
|
||||
For data-aware weight compression refer to the following `example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama>`__.
|
||||
|
||||
.. note::
|
||||
The example below shows data-free 4-bit weight quantization
|
||||
applied on top of OpenVINO IR. Before trying the example, make sure Optimum Intel
|
||||
is installed in your environment by running the following command:
|
||||
|
||||
OpenVINO also supports 4-bit models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized
|
||||
with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it.
|
||||
.. code-block:: python
|
||||
|
||||
pip install optimum[openvino,nncf]
|
||||
|
||||
The first example loads a pre-trained Hugging Face model using the Optimum Intel API,
|
||||
compresses it to INT4 using NNCF, and then executes inference with a text phrase.
|
||||
|
||||
If the model comes from Hugging Face and is supported by Optimum, it can
|
||||
be easier to use the Optimum Intel API to perform weight compression. The compression
|
||||
type is specified when the model is loaded using the ``load_in_8bit=True`` or ``load_in_4bit=True`` parameter.
|
||||
The second example uses the Weight Compression API from Optimum Intel instead of NNCF
|
||||
to compress the model to INT8.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: OpenVINO
|
||||
:sync: openvino
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from nncf import compress_weights, CompressWeightsMode
|
||||
from optimum.intel.openvino import OVModelForCausalLM
|
||||
from transformers import AutoTokenizer, pipeline
|
||||
|
||||
# Load model from Hugging Face
|
||||
model_id = "HuggingFaceH4/zephyr-7b-beta"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||||
|
||||
# Compress to INT4 Symmetric
|
||||
model.model = compress_weights(model.model, mode=CompressWeightsMode.INT4_SYM)
|
||||
|
||||
# Inference
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||||
phrase = "The weather is"
|
||||
results = pipe(phrase)
|
||||
print(results)
|
||||
|
||||
.. tab-item:: Optimum-Intel
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from optimum.intel.openvino import OVModelForCausalLM
|
||||
from transformers import AutoTokenizer, pipeline
|
||||
|
||||
# Load and compress model from Hugging Face
|
||||
model_id = "HuggingFaceH4/zephyr-7b-beta"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True, load_in_8bit=True)
|
||||
|
||||
# Inference
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||||
phrase = "The weather is"
|
||||
results = pipe(phrase)
|
||||
print(results)
|
||||
|
||||
Exporting and Loading Compressed Models
|
||||
########################################
|
||||
|
||||
Once a model has been compressed with NNCF or Optimum Intel,
|
||||
it can be saved and exported to use in a future session or in a
|
||||
deployment environment. The compression process takes a while,
|
||||
so it is preferable to compress the model once, save it, and then
|
||||
load the compressed model later for faster time to first inference.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Save compressed model for faster loading later
|
||||
model.save_pretrained("zephyr-7b-beta-int4-sym-ov")
|
||||
tokenizer.save_pretrained("zephyr-7b-beta-int4-sym-ov")
|
||||
|
||||
# Load a saved model
|
||||
model = OVModelForCausalLM.from_pretrained("zephyr-7b-beta-int4-sym-ov")
|
||||
tokenizer = AutoTokenizer.from_pretrained("zephyr-7b-beta-int4-sym-ov")
|
||||
|
||||
GPTQ Models
|
||||
############
|
||||
|
||||
OpenVINO also supports 4-bit models from Hugging Face
|
||||
`Transformers <https://github.com/huggingface/transformers>`__ library optimized
|
||||
with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. In this case, there is no
|
||||
need for an additional model optimization step because model conversion will
|
||||
automatically preserve the INT4 optimization results, allowing model inference to benefit from it.
|
||||
|
||||
A compression example using a GPTQ model is shown below.
|
||||
Make sure to install GPTQ dependencies by running the following command:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pip install optimum[openvino] auto-gptq
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from optimum.intel.openvino import OVModelForCausalLM
|
||||
from transformers import AutoTokenizer, pipeline
|
||||
|
||||
# Load model from Hugging Face already optimized with GPTQ
|
||||
model_id = "TheBloke/Llama-2-7B-Chat-GPTQ"
|
||||
model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||||
|
||||
# Inference
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||||
phrase = "The weather is"
|
||||
results = pipe(phrase)
|
||||
print(results)
|
||||
|
||||
An `example of a model <https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ>`__ that has been optimized using GPTQ.
|
||||
|
||||
Compression Metrics Examples
|
||||
########################################
|
||||
|
||||
The table below shows examples of text-generation Language Models with different optimization settings in a data-free setup, where no dataset is used at the optimization step.
|
||||
The Perplexity metric is measured on the `Lambada OpenAI dataset <https://github.com/openai/gpt-2/issues/131#issuecomment-497136199>`__.
|
||||
The Perplexity metric is a measurement of response accuracy, where a higher complexity score indicates a lower accuracy.
|
||||
It is measured on the `Lambada OpenAI dataset <https://github.com/openai/gpt-2/issues/131#issuecomment-497136199>`__.
|
||||
|
||||
.. list-table::
|
||||
:widths: 40 55 25 25
|
||||
|
|
@ -188,23 +387,21 @@ The following table shows accuracy metric in a data-aware 4-bit weight quantizat
|
|||
|
||||
|
||||
\*Perplexity metric in both tables was measured without the Dynamic Quantization feature enabled in the OpenVINO runtime.
|
||||
|
||||
|
||||
|
||||
Auto-tuning of Weight Compression Parameters
|
||||
############################################
|
||||
|
||||
To find the optimal weight compression parameters for a particular model, refer to the `example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__ , where weight compression parameters are being searched from the subset of values. To speed up the search, a self-designed
|
||||
validation pipeline called `WhoWhatBench <https://github.com/openvinotoolkit/openvino.genai/tree/master/llm_bench/python/who_what_benchmark>`__ is used.
|
||||
To find the optimal weight compression parameters for a particular model, refer to the `example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__ , where weight compression parameters are being searched from the subset of values. To speed up the search, a self-designed
|
||||
validation pipeline called `WhoWhatBench <https://github.com/openvinotoolkit/openvino.genai/tree/master/llm_bench/python/who_what_benchmark>`__ is used.
|
||||
The pipeline can quickly evaluate the changes in the accuracy of the optimized model compared to the baseline.
|
||||
|
||||
|
||||
Additional Resources
|
||||
####################
|
||||
|
||||
- `Data-aware Weight Compression Example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama>`__
|
||||
- `Tune Weight Compression Parameters Example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/tiny_llama_find_hyperparams>`__
|
||||
- `WhoWhatBench <https://github.com/openvinotoolkit/openvino.genai/tree/master/llm_bench/python/who_what_benchmark>`__
|
||||
- `OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>`__: Repository containing example pipelines that implement image and text generation tasks. It also provides a tool to benchmark LLMs.
|
||||
- `WhoWhatBench <https://github.com/openvinotoolkit/openvino.genai/tree/master/llm_bench/python/who_what_benchmark>`__
|
||||
- `NNCF GitHub <https://github.com/openvinotoolkit/nncf>`__
|
||||
- :doc:`Post-training Quantization <ptq_introduction>`
|
||||
- :doc:`Training-time Optimization <tmo_introduction>`
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ from the application code to OpenVINO and all related internal work is hidden fr
|
|||
|
||||
There are three methods of turning an OpenVINO model into a stateful one:
|
||||
|
||||
* :doc:`Optimum-Intel<gen_ai_guide>` - the most user-friendly option. All necessary optimizations
|
||||
* :doc:`Optimum-Intel<llm_inference>` - the most user-friendly option. All necessary optimizations
|
||||
are recognized and applied automatically. The drawback is, the tool does not work with all
|
||||
models.
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ and you have three ways to do it:
|
|||
|
||||
* `Optimum-Intel <https://github.com/huggingface/optimum-intel>`__ - an automated solution
|
||||
applicable to a selection of models (not covered by this article, for a usage guide
|
||||
refer to the :doc:`Optimize and Deploy Generative AI Models <gen_ai_guide>` article).
|
||||
refer to the :doc:`LLM Inference with Hugging Face and Optimum Intel <llm_inference>` article).
|
||||
* :ref:`MakeStateful transformation <ov_ug_make_stateful>` - to choose which pairs of
|
||||
Parameter and Result to replace.
|
||||
* :ref:`LowLatency2 transformation <ov_ug_low_latency>` - to detect and replace Parameter
|
||||
|
|
|
|||
|
|
@ -90,7 +90,7 @@ OpenVINO 2023.2
|
|||
Optimize generation of the graph model with PyTorch 2.0 torch.compile() backend
|
||||
|
||||
.. grid-item-card:: Optimize and Deploy Generative AI
|
||||
:link: gen_ai_guide
|
||||
:link: native_vs_hugging_face_api
|
||||
:link-alt: gen ai
|
||||
:link-type: doc
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue