1544 lines
57 KiB
ReStructuredText
1544 lines
57 KiB
ReStructuredText
Create an LLM-powered RAG system using OpenVINO
|
||
===============================================
|
||
|
||
**Retrieval-augmented generation (RAG)** is a technique for augmenting
|
||
LLM knowledge with additional, often private or real-time, data. LLMs
|
||
can reason about wide-ranging topics, but their knowledge is limited to
|
||
the public data up to a specific point in time that they were trained
|
||
on. If you want to build AI applications that can reason about private
|
||
data or data introduced after a model’s cutoff date, you need to augment
|
||
the knowledge of the model with the specific information it needs. The
|
||
process of bringing the appropriate information and inserting it into
|
||
the model prompt is known as Retrieval Augmented Generation (RAG).
|
||
|
||
`LangChain <https://python.langchain.com/docs/get_started/introduction>`__
|
||
is a framework for developing applications powered by language models.
|
||
It has a number of components specifically designed to help build RAG
|
||
applications. In this tutorial, we’ll build a simple question-answering
|
||
application over a Markdown or CSV data source.
|
||
|
||
The tutorial consists of the following steps:
|
||
|
||
- Install prerequisites
|
||
- Download and convert the model from a public source using the
|
||
`OpenVINO integration with Hugging Face
|
||
Optimum <https://huggingface.co/blog/openvino>`__.
|
||
- Compress model weights to 4-bit or 8-bit data types using
|
||
`NNCF <https://github.com/openvinotoolkit/nncf>`__
|
||
- Create a RAG chain pipeline
|
||
- Run chat pipeline
|
||
|
||
Table of contents:
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
- `Prerequisites <#prerequisites>`__
|
||
- `Select model for inference <#select-model-for-inference>`__
|
||
- `login to huggingfacehub to get access to pretrained
|
||
model <#login-to-huggingfacehub-to-get-access-to-pretrained-model>`__
|
||
- `Convert model <#convert-model>`__
|
||
|
||
- `Convert LLM model <#convert-llm-model>`__
|
||
|
||
- `Compress model weights <#compress-model-weights>`__
|
||
|
||
- `Weights Compression using Optimum
|
||
Intel <#weights-compression-using-optimum-intel>`__
|
||
- `Weights Compression using
|
||
NNCF <#weights-compression-using-nncf>`__
|
||
- `Convert embedding model <#convert-embedding-model>`__
|
||
|
||
- `Select device for inference and model
|
||
variant <#select-device-for-inference-and-model-variant>`__
|
||
|
||
- `Select device for embedding model
|
||
inference <#select-device-for-embedding-model-inference>`__
|
||
- `Select device for LLM model
|
||
inference <#select-device-for-llm-model-inference>`__
|
||
|
||
- `Load model <#load-model>`__
|
||
|
||
- `Load embedding model <#load-embedding-model>`__
|
||
- `Load LLM model <#load-llm-model>`__
|
||
|
||
- `Run QA over Document <#run-qa-over-document>`__
|
||
|
||
Prerequisites
|
||
-------------
|
||
|
||
|
||
|
||
Install required dependencies
|
||
|
||
.. code:: ipython3
|
||
|
||
%pip uninstall -q -y openvino-dev openvino openvino-nightly optimum optimum-intel
|
||
%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu\
|
||
"git+https://github.com/huggingface/optimum-intel.git"\
|
||
"git+https://github.com/openvinotoolkit/nncf.git"\
|
||
"datasets"\
|
||
"accelerate"\
|
||
"openvino-nightly"\
|
||
"gradio"\
|
||
"onnx" "chromadb" "sentence_transformers" "langchain>=0.1.7" "langchainhub" "transformers>=4.37.0" "unstructured" "scikit-learn" "python-docx" "pdfminer.six" "bitsandbytes"
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
WARNING: Skipping openvino-dev as it is not installed.
|
||
WARNING: Skipping openvino as it is not installed.
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
|
||
[notice] A new release of pip is available: 23.3.2 -> 24.0
|
||
[notice] To update, run: pip install --upgrade pip
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
|
||
|
||
Select model for inference
|
||
--------------------------
|
||
|
||
|
||
|
||
The tutorial supports different models, you can select one from the
|
||
provided options to compare the quality of open source LLM solutions.
|
||
>\ **Note**: conversion of some models can require additional actions
|
||
from user side and at least 64GB RAM for conversion.
|
||
|
||
The available embedding model options are:
|
||
|
||
- **all-mpnet-base-v2(All)** - This is a
|
||
`sentence-transformers <https://huggingface.co/sentence-transformers>`__
|
||
model: It maps sentences & paragraphs to a 768 dimensional dense
|
||
vector space and can be used for tasks like clustering or semantic
|
||
search. More details about model can be found in `model
|
||
card <https://huggingface.co/sentence-transformers/all-mpnet-base-v2>`__
|
||
- **text2vec-large-chinese(Chinese)** - This is a
|
||
`CoSENT <https://github.com/bojone/CoSENT>`__ model. It can be used
|
||
for tasks like sentence embeddings, text matching or semantic search.
|
||
More details about model can be found in `model
|
||
card <https://huggingface.co/GanymedeNil/text2vec-base-chinese>`__
|
||
|
||
The available LLM model options are:
|
||
|
||
- **tiny-llama-1b-chat** - This is the chat model finetuned on top of
|
||
`TinyLlama/TinyLlama-1.1B-intermediate-step-955k-2T <https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-955k-token-2T>`__.
|
||
The TinyLlama project aims to pretrain a 1.1B Llama model on 3
|
||
trillion tokens with the adoption of the same architecture and
|
||
tokenizer as Llama 2. This means TinyLlama can be plugged and played
|
||
in many open-source projects built upon Llama. Besides, TinyLlama is
|
||
compact with only 1.1B parameters. This compactness allows it to
|
||
cater to a multitude of applications demanding a restricted
|
||
computation and memory footprint. More details about model can be
|
||
found in `model
|
||
card <https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6>`__
|
||
- **gemma-2b-it** - Gemma is a family of lightweight, state-of-the-art
|
||
open models from Google, built from the same research and technology
|
||
used to create the Gemini models. They are text-to-text, decoder-only
|
||
large language models, available in English, with open weights,
|
||
pre-trained variants, and instruction-tuned variants. Gemma models
|
||
are well-suited for a variety of text generation tasks, including
|
||
question answering, summarization, and reasoning. This model is
|
||
instruction-tuned version of 2B parameters model. More details about
|
||
model can be found in `model
|
||
card <https://huggingface.co/google/gemma-2b-it>`__. >\ **Note**: run
|
||
model with demo, you will need to accept license agreement. >You must
|
||
be a registered user in Hugging Face Hub. Please visit `HuggingFace
|
||
model card <https://huggingface.co/google/gemma-2b-it>`__, carefully
|
||
read terms of usage and click accept button. You will need to use an
|
||
access token for the code below to run. For more information on
|
||
access tokens, refer to `this section of the
|
||
documentation <https://huggingface.co/docs/hub/security-tokens>`__.
|
||
>You can login on Hugging Face Hub in notebook environment, using
|
||
following code:
|
||
|
||
.. code:: python
|
||
|
||
## login to huggingfacehub to get access to pretrained model
|
||
|
||
from huggingface_hub import notebook_login, whoami
|
||
|
||
try:
|
||
whoami()
|
||
print('Authorization token already provided')
|
||
except OSError:
|
||
notebook_login()
|
||
|
||
- **mini-cpm-2b-dpo** - MiniCPM is an End-Size LLM developed by
|
||
ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding
|
||
embeddings. After Direct Preference Optimization (DPO) fine-tuning,
|
||
MiniCPM outperforms many popular 7b, 13b and 70b models. More details
|
||
can be found in
|
||
`model_card <https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16>`__.
|
||
- **red-pajama-3b-chat** - A 2.8B parameter pre-trained language model
|
||
based on GPT-NEOX architecture. It was developed by Together Computer
|
||
and leaders from the open-source AI community. The model is
|
||
fine-tuned on OASST1 and Dolly2 datasets to enhance chatting ability.
|
||
More details about model can be found in `HuggingFace model
|
||
card <https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1>`__.
|
||
- **gemma-7b-it** - Gemma is a family of lightweight, state-of-the-art
|
||
open models from Google, built from the same research and technology
|
||
used to create the Gemini models. They are text-to-text, decoder-only
|
||
large language models, available in English, with open weights,
|
||
pre-trained variants, and instruction-tuned variants. Gemma models
|
||
are well-suited for a variety of text generation tasks, including
|
||
question answering, summarization, and reasoning. This model is
|
||
instruction-tuned version of 7B parameters model. More details about
|
||
model can be found in `model
|
||
card <https://huggingface.co/google/gemma-7b-it>`__. >\ **Note**: run
|
||
model with demo, you will need to accept license agreement. >You must
|
||
be a registered user in Hugging Face Hub. Please visit `HuggingFace
|
||
model card <https://huggingface.co/google/gemma-7b-it>`__, carefully
|
||
read terms of usage and click accept button. You will need to use an
|
||
access token for the code below to run. For more information on
|
||
access tokens, refer to `this section of the
|
||
documentation <https://huggingface.co/docs/hub/security-tokens>`__.
|
||
>You can login on Hugging Face Hub in notebook environment, using
|
||
following code:
|
||
|
||
.. code:: python
|
||
|
||
## login to huggingfacehub to get access to pretrained model
|
||
|
||
from huggingface_hub import notebook_login, whoami
|
||
|
||
try:
|
||
whoami()
|
||
print('Authorization token already provided')
|
||
except OSError:
|
||
notebook_login()
|
||
|
||
- **llama-2-7b-chat** - LLama 2 is the second generation of LLama
|
||
models developed by Meta. Llama 2 is a collection of pre-trained and
|
||
fine-tuned generative text models ranging in scale from 7 billion to
|
||
70 billion parameters. llama-2-7b-chat is 7 billions parameters
|
||
version of LLama 2 finetuned and optimized for dialogue use case.
|
||
More details about model can be found in the
|
||
`paper <https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/>`__,
|
||
`repository <https://github.com/facebookresearch/llama>`__ and
|
||
`HuggingFace model
|
||
card <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`__.
|
||
>\ **Note**: run model with demo, you will need to accept license
|
||
agreement. >You must be a registered user in Hugging Face Hub.
|
||
Please visit `HuggingFace model
|
||
card <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`__,
|
||
carefully read terms of usage and click accept button. You will need
|
||
to use an access token for the code below to run. For more
|
||
information on access tokens, refer to `this section of the
|
||
documentation <https://huggingface.co/docs/hub/security-tokens>`__.
|
||
>You can login on Hugging Face Hub in notebook environment, using
|
||
following code:
|
||
|
||
.. code:: python
|
||
|
||
## login to huggingfacehub to get access to pretrained model
|
||
|
||
from huggingface_hub import notebook_login, whoami
|
||
|
||
try:
|
||
whoami()
|
||
print('Authorization token already provided')
|
||
except OSError:
|
||
notebook_login()
|
||
|
||
- **qwen1.5-0.5b-chat/qwen1.5-1.8b-chat/qwen1.5-7b-chat** - Qwen1.5 is
|
||
the beta version of Qwen2, a transformer-based decoder-only language
|
||
model pretrained on a large amount of data. Qwen1.5 is a language
|
||
model series including decoder language models of different model
|
||
sizes. It is based on the Transformer architecture with SwiGLU
|
||
activation, attention QKV bias, group query attention, mixture of
|
||
sliding window attention and full attention. You can find more
|
||
details about model in the `model
|
||
repository <https://huggingface.co/Qwen>`__.
|
||
- **qwen-7b-chat** - Qwen-7B is the 7B-parameter version of the large
|
||
language model series, Qwen (abbr. Tongyi Qianwen), proposed by
|
||
Alibaba Cloud. Qwen-7B is a Transformer-based large language model,
|
||
which is pretrained on a large volume of data, including web texts,
|
||
books, codes, etc. For more details about Qwen, please refer to the
|
||
`GitHub <https://github.com/QwenLM/Qwen>`__ code repository.
|
||
- **mpt-7b-chat** - MPT-7B is part of the family of
|
||
MosaicPretrainedTransformer (MPT) models, which use a modified
|
||
transformer architecture optimized for efficient training and
|
||
inference. These architectural changes include performance-optimized
|
||
layer implementations and the elimination of context length limits by
|
||
replacing positional embeddings with Attention with Linear Biases
|
||
(`ALiBi <https://arxiv.org/abs/2108.12409>`__). Thanks to these
|
||
modifications, MPT models can be trained with high throughput
|
||
efficiency and stable convergence. MPT-7B-chat is a chatbot-like
|
||
model for dialogue generation. It was built by finetuning MPT-7B on
|
||
the
|
||
`ShareGPT-Vicuna <https://huggingface.co/datasets/jeffwan/sharegpt_vicuna>`__,
|
||
`HC3 <https://huggingface.co/datasets/Hello-SimpleAI/HC3>`__,
|
||
`Alpaca <https://huggingface.co/datasets/tatsu-lab/alpaca>`__,
|
||
`HH-RLHF <https://huggingface.co/datasets/Anthropic/hh-rlhf>`__, and
|
||
`Evol-Instruct <https://huggingface.co/datasets/victor123/evol_instruct_70k>`__
|
||
datasets. More details about the model can be found in `blog
|
||
post <https://www.mosaicml.com/blog/mpt-7b>`__,
|
||
`repository <https://github.com/mosaicml/llm-foundry/>`__ and
|
||
`HuggingFace model
|
||
card <https://huggingface.co/mosaicml/mpt-7b-chat>`__.
|
||
- **chatglm3-6b** - ChatGLM3-6B is the latest open-source model in the
|
||
ChatGLM series. While retaining many excellent features such as
|
||
smooth dialogue and low deployment threshold from the previous two
|
||
generations, ChatGLM3-6B employs a more diverse training dataset,
|
||
more sufficient training steps, and a more reasonable training
|
||
strategy. ChatGLM3-6B adopts a newly designed `Prompt
|
||
format <https://github.com/THUDM/ChatGLM3/blob/main/PROMPT_en.md>`__,
|
||
in addition to the normal multi-turn dialogue. You can find more
|
||
details about model in the `model
|
||
card <https://huggingface.co/THUDM/chatglm3-6b>`__
|
||
- **mistral-7b** - The Mistral-7B-v0.1 Large Language Model (LLM) is a
|
||
pretrained generative text model with 7 billion parameters. You can
|
||
find more details about model in the `model
|
||
card <https://huggingface.co/mistralai/Mistral-7B-v0.1>`__,
|
||
`paper <https://arxiv.org/abs/2310.06825>`__ and `release blog
|
||
post <https://mistral.ai/news/announcing-mistral-7b/>`__.
|
||
- **zephyr-7b-beta** - Zephyr is a series of language models that are
|
||
trained to act as helpful assistants. Zephyr-7B-beta is the second
|
||
model in the series, and is a fine-tuned version of
|
||
`mistralai/Mistral-7B-v0.1 <https://huggingface.co/mistralai/Mistral-7B-v0.1>`__
|
||
that was trained on on a mix of publicly available, synthetic
|
||
datasets using `Direct Preference Optimization
|
||
(DPO) <https://arxiv.org/abs/2305.18290>`__. You can find more
|
||
details about model in `technical
|
||
report <https://arxiv.org/abs/2310.16944>`__ and `HuggingFace model
|
||
card <https://huggingface.co/HuggingFaceH4/zephyr-7b-beta>`__.
|
||
- **neural-chat-7b-v3-1** - Mistral-7b model fine-tuned using Intel
|
||
Gaudi. The model fine-tuned on the open source dataset
|
||
`Open-Orca/SlimOrca <https://huggingface.co/datasets/Open-Orca/SlimOrca>`__
|
||
and aligned with `Direct Preference Optimization (DPO)
|
||
algorithm <https://arxiv.org/abs/2305.18290>`__. More details can be
|
||
found in `model
|
||
card <https://huggingface.co/Intel/neural-chat-7b-v3-1>`__ and `blog
|
||
post <https://medium.com/@NeuralCompressor/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3>`__.
|
||
- **notus-7b-v1** - Notus is a collection of fine-tuned models using
|
||
`Direct Preference Optimization
|
||
(DPO) <https://arxiv.org/abs/2305.18290>`__. and related
|
||
`RLHF <https://huggingface.co/blog/rlhf>`__ techniques. This model is
|
||
the first version, fine-tuned with DPO over zephyr-7b-sft. Following
|
||
a data-first approach, the only difference between Notus-7B-v1 and
|
||
Zephyr-7B-beta is the preference dataset used for dDPO. Proposed
|
||
approach for dataset creation helps to effectively fine-tune Notus-7b
|
||
that surpasses Zephyr-7B-beta and Claude 2 on
|
||
`AlpacaEval <https://tatsu-lab.github.io/alpaca_eval/>`__. More
|
||
details about model can be found in `model
|
||
card <https://huggingface.co/argilla/notus-7b-v1>`__.
|
||
- **youri-7b-chat** - Youri-7b-chat is a Llama2 based model. `Rinna
|
||
Co., Ltd. <https://rinna.co.jp/>`__ conducted further pre-training
|
||
for the Llama2 model with a mixture of English and Japanese datasets
|
||
to improve Japanese task capability. The model is publicly released
|
||
on Hugging Face hub. You can find detailed information at the
|
||
`rinna/youri-7b-chat project
|
||
page <https://huggingface.co/rinna/youri-7b>`__.
|
||
- **baichuan2-7b-chat** - Baichuan 2 is the new generation of
|
||
large-scale open-source language models launched by `Baichuan
|
||
Intelligence inc <https://www.baichuan-ai.com/home>`__. It is trained
|
||
on a high-quality corpus with 2.6 trillion tokens and has achieved
|
||
the best performance in authoritative Chinese and English benchmarks
|
||
of the same size.
|
||
- **internlm2-chat-1.8b** - InternLM2 is the second generation InternLM
|
||
series. Compared to the previous generation model, it shows
|
||
significant improvements in various capabilities, including
|
||
reasoning, mathematics, and coding. More details about model can be
|
||
found in `model repository <https://huggingface.co/internlm>`__.
|
||
|
||
.. code:: ipython3
|
||
|
||
from pathlib import Path
|
||
from optimum.intel.openvino import OVModelForCausalLM
|
||
import openvino as ov
|
||
import torch
|
||
import nncf
|
||
import logging
|
||
import shutil
|
||
import gc
|
||
import ipywidgets as widgets
|
||
from transformers import (
|
||
AutoModelForCausalLM,
|
||
AutoModel,
|
||
AutoTokenizer,
|
||
AutoConfig,
|
||
TextIteratorStreamer,
|
||
pipeline,
|
||
StoppingCriteria,
|
||
StoppingCriteriaList,
|
||
)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
2024-03-06 07:05:19.617312: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
|
||
2024-03-06 07:05:19.620814: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
|
||
2024-03-06 07:05:19.663621: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
|
||
2024-03-06 07:05:19.663653: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
|
||
2024-03-06 07:05:19.663683: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
|
||
2024-03-06 07:05:19.671963: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
|
||
2024-03-06 07:05:19.673938: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
|
||
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
|
||
2024-03-06 07:05:20.726709: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
|
||
|
||
|
||
Convert model
|
||
-------------
|
||
|
||
|
||
|
||
Convert LLM model
|
||
~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
from config import SUPPORTED_EMBEDDING_MODELS, SUPPORTED_LLM_MODELS
|
||
|
||
model_languages = list(SUPPORTED_LLM_MODELS)
|
||
|
||
model_language = widgets.Dropdown(
|
||
options=model_languages,
|
||
value=model_languages[0],
|
||
description="Model Language:",
|
||
disabled=False,
|
||
)
|
||
|
||
model_language
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Model Language:', options=('English', 'Chinese', 'Japanese'), value='English')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
llm_model_ids = list(SUPPORTED_LLM_MODELS[model_language.value])
|
||
|
||
llm_model_id = widgets.Dropdown(
|
||
options=llm_model_ids,
|
||
value=llm_model_ids[4],
|
||
description="Model:",
|
||
disabled=False,
|
||
)
|
||
|
||
llm_model_id
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Model:', options=('tiny-llama-1b-chat', 'gemma-2b-it', 'red-pajama-3b-chat', 'gemma-7b-i…
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
llm_model_configuration = SUPPORTED_LLM_MODELS[model_language.value][llm_model_id.value]
|
||
print(f"Selected LLM model {llm_model_id.value}")
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Selected LLM model tiny-llama-1b-chat
|
||
|
||
|
||
Optimum Intel can be used to load optimized models from the `Hugging
|
||
Face Hub <https://huggingface.co/docs/optimum/intel/hf.co/models>`__ and
|
||
create pipelines to run an inference with OpenVINO Runtime using Hugging
|
||
Face APIs. The Optimum Inference models are API compatible with Hugging
|
||
Face Transformers models. This means we just need to replace
|
||
``AutoModelForXxx`` class with the corresponding ``OVModelForXxx``
|
||
class.
|
||
|
||
Below is an example of the RedPajama model
|
||
|
||
.. code:: diff
|
||
|
||
-from transformers import AutoModelForCausalLM
|
||
+from optimum.intel.openvino import OVModelForCausalLM
|
||
from transformers import AutoTokenizer, pipeline
|
||
|
||
model_id = "togethercomputer/RedPajama-INCITE-Chat-3B-v1"
|
||
-model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
+model = OVModelForCausalLM.from_pretrained(model_id, export=True)
|
||
|
||
Model class initialization starts with calling ``from_pretrained``
|
||
method. When downloading and converting Transformers model, the
|
||
parameter ``export=True`` should be added. We can save the converted
|
||
model for the next usage with the ``save_pretrained`` method. Tokenizer
|
||
class and pipelines API are compatible with Optimum models.
|
||
|
||
To optimize the generation process and use memory more efficiently, the
|
||
``use_cache=True`` option is enabled. Since the output side is
|
||
auto-regressive, an output token hidden state remains the same once
|
||
computed for every further generation step. Therefore, recomputing it
|
||
every time you want to generate a new token seems wasteful. With the
|
||
cache, the model saves the hidden state once it has been computed. The
|
||
model only computes the one for the most recently generated output token
|
||
at each time step, re-using the saved ones for hidden tokens. This
|
||
reduces the generation complexity from :math:`O(n^3)` to :math:`O(n^2)`
|
||
for a transformer model. More details about how it works can be found in
|
||
this
|
||
`article <https://scale.com/blog/pytorch-improvements#Text%20Translation>`__.
|
||
With this option, the model gets the previous step’s hidden states
|
||
(cached attention keys and values) as input and additionally provides
|
||
hidden states for the current step as output. It means for all next
|
||
iterations, it is enough to provide only a new token obtained from the
|
||
previous step and cached key values to get the next token prediction.
|
||
|
||
In our case, MPT, Qwen and ChatGLM model currently is not covered by
|
||
Optimum Intel, we will convert it manually and create wrapper compatible
|
||
with Optimum Intel.
|
||
|
||
Compress model weights
|
||
----------------------
|
||
|
||
|
||
|
||
The Weights Compression algorithm is aimed at compressing the weights of
|
||
the models and can be used to optimize the model footprint and
|
||
performance of large models where the size of weights is relatively
|
||
larger than the size of activations, for example, Large Language Models
|
||
(LLM). Compared to INT8 compression, INT4 compression improves
|
||
performance even more, but introduces a minor drop in prediction
|
||
quality.
|
||
|
||
Weights Compression using Optimum Intel
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
Optimum Intel supports weight compression via NNCF out of the box. For
|
||
8-bit compression we pass ``load_in_8bit=True`` to ``from_pretrained()``
|
||
method of ``OVModelForCausalLM``. For 4 bit compression we provide
|
||
``quantization_config=OVWeightQuantizationConfig(bits=4, ...)`` argument
|
||
containing number of bits and other compression parameters. We will
|
||
consider how to do it on RedPajama, LLAMA and Zephyr examples.
|
||
|
||
**Note**: There may be no speedup for INT4/INT8 compressed models on
|
||
dGPU.
|
||
|
||
Weights Compression using NNCF
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
You also can perform weights compression for OpenVINO models using NNCF
|
||
directly. ``nncf.compress_weights`` function accepts OpenVINO model
|
||
instance and compresses its weights for Linear and Embedding layers. We
|
||
will consider this variant based on MPT model.
|
||
|
||
**Note**: This tutorial involves conversion model for FP16 and
|
||
INT4/INT8 weights compression scenarios. It may be memory and
|
||
time-consuming in the first run. You can manually control the
|
||
compression precision below.
|
||
|
||
.. code:: ipython3
|
||
|
||
from IPython.display import display
|
||
|
||
prepare_int4_model = widgets.Checkbox(
|
||
value=True,
|
||
description="Prepare INT4 model",
|
||
disabled=False,
|
||
)
|
||
prepare_int8_model = widgets.Checkbox(
|
||
value=False,
|
||
description="Prepare INT8 model",
|
||
disabled=False,
|
||
)
|
||
prepare_fp16_model = widgets.Checkbox(
|
||
value=False,
|
||
description="Prepare FP16 model",
|
||
disabled=False,
|
||
)
|
||
|
||
display(prepare_int4_model)
|
||
display(prepare_int8_model)
|
||
display(prepare_fp16_model)
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Checkbox(value=True, description='Prepare INT4 model')
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Checkbox(value=False, description='Prepare INT8 model')
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Checkbox(value=False, description='Prepare FP16 model')
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
from optimum.intel import OVWeightQuantizationConfig
|
||
from converter import converters, register_configs
|
||
|
||
register_configs()
|
||
|
||
nncf.set_log_level(logging.ERROR)
|
||
|
||
pt_model_id = llm_model_configuration["model_id"]
|
||
pt_model_name = llm_model_id.value.split("-")[0]
|
||
model_type = AutoConfig.from_pretrained(pt_model_id, trust_remote_code=True).model_type
|
||
fp16_model_dir = Path(llm_model_id.value) / "FP16"
|
||
int8_model_dir = Path(llm_model_id.value) / "INT8_compressed_weights"
|
||
int4_model_dir = Path(llm_model_id.value) / "INT4_compressed_weights"
|
||
|
||
|
||
def convert_to_fp16():
|
||
if (fp16_model_dir / "openvino_model.xml").exists():
|
||
return
|
||
if not llm_model_configuration["remote"]:
|
||
remote_code = llm_model_configuration.get("remote_code", False)
|
||
model_kwargs = {}
|
||
if remote_code:
|
||
model_kwargs = {
|
||
"trust_remote_code": True,
|
||
"config": AutoConfig.from_pretrained(pt_model_id, trust_remote_code=True)
|
||
}
|
||
ov_model = OVModelForCausalLM.from_pretrained(
|
||
pt_model_id, export=True, compile=False, load_in_8bit=False, **model_kwargs
|
||
)
|
||
ov_model.half()
|
||
ov_model.save_pretrained(fp16_model_dir)
|
||
del ov_model
|
||
else:
|
||
model_kwargs = {}
|
||
if "revision" in llm_model_configuration:
|
||
model_kwargs["revision"] = llm_model_configuration["revision"]
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
llm_model_configuration["model_id"],
|
||
torch_dtype=torch.float32,
|
||
trust_remote_code=True,
|
||
**model_kwargs
|
||
)
|
||
converters[pt_model_name](model, fp16_model_dir)
|
||
del model
|
||
gc.collect()
|
||
|
||
|
||
def convert_to_int8():
|
||
if (int8_model_dir / "openvino_model.xml").exists():
|
||
return
|
||
int8_model_dir.mkdir(parents=True, exist_ok=True)
|
||
if not llm_model_configuration["remote"]:
|
||
remote_code = llm_model_configuration.get("remote_code", False)
|
||
model_kwargs = {}
|
||
if remote_code:
|
||
model_kwargs = {
|
||
"trust_remote_code": True,
|
||
"config": AutoConfig.from_pretrained(pt_model_id, trust_remote_code=True)
|
||
}
|
||
ov_model = OVModelForCausalLM.from_pretrained(
|
||
pt_model_id, export=True, compile=False, load_in_8bit=True, **model_kwargs
|
||
)
|
||
ov_model.save_pretrained(int8_model_dir)
|
||
del ov_model
|
||
else:
|
||
convert_to_fp16()
|
||
ov_model = ov.Core().read_model(fp16_model_dir / "openvino_model.xml")
|
||
shutil.copy(fp16_model_dir / "config.json", int8_model_dir / "config.json")
|
||
configuration_file = fp16_model_dir / f"configuration_{model_type}.py"
|
||
if configuration_file.exists():
|
||
shutil.copy(
|
||
configuration_file, int8_model_dir / f"configuration_{model_type}.py"
|
||
)
|
||
compressed_model = nncf.compress_weights(ov_model)
|
||
ov.save_model(compressed_model, int8_model_dir / "openvino_model.xml")
|
||
del ov_model
|
||
del compressed_model
|
||
gc.collect()
|
||
|
||
|
||
def convert_to_int4():
|
||
compression_configs = {
|
||
"zephyr-7b-beta": {
|
||
"sym": True,
|
||
"group_size": 64,
|
||
"ratio": 0.6,
|
||
},
|
||
"mistral-7b": {
|
||
"sym": True,
|
||
"group_size": 64,
|
||
"ratio": 0.6,
|
||
},
|
||
"minicpm-2b-dpo": {
|
||
"sym": True,
|
||
"group_size": 64,
|
||
"ratio": 0.6,
|
||
},
|
||
"notus-7b-v1": {
|
||
"sym": True,
|
||
"group_size": 64,
|
||
"ratio": 0.6,
|
||
},
|
||
"neural-chat-7b-v3-1": {
|
||
"sym": True,
|
||
"group_size": 64,
|
||
"ratio": 0.6,
|
||
},
|
||
"llama-2-chat-7b": {
|
||
"sym": True,
|
||
"group_size": 128,
|
||
"ratio": 0.8,
|
||
},
|
||
"chatglm2-6b": {
|
||
"sym": True,
|
||
"group_size": 128,
|
||
"ratio": 0.72,
|
||
},
|
||
"qwen-7b-chat": {
|
||
"sym": True,
|
||
"group_size": 128,
|
||
"ratio": 0.6
|
||
},
|
||
'red-pajama-3b-chat': {
|
||
"sym": False,
|
||
"group_size": 128,
|
||
"ratio": 0.5,
|
||
},
|
||
"default": {
|
||
"sym": False,
|
||
"group_size": 128,
|
||
"ratio": 0.8,
|
||
},
|
||
}
|
||
|
||
model_compression_params = compression_configs.get(
|
||
llm_model_id.value, compression_configs["default"]
|
||
)
|
||
if (int4_model_dir / "openvino_model.xml").exists():
|
||
return
|
||
int4_model_dir.mkdir(parents=True, exist_ok=True)
|
||
if not llm_model_configuration["remote"]:
|
||
remote_code = llm_model_configuration.get("remote_code", False)
|
||
model_kwargs = {}
|
||
if remote_code:
|
||
model_kwargs = {
|
||
"trust_remote_code" : True,
|
||
"config": AutoConfig.from_pretrained(pt_model_id, trust_remote_code=True)
|
||
}
|
||
ov_model = OVModelForCausalLM.from_pretrained(
|
||
pt_model_id, export=True, compile=False,
|
||
quantization_config=OVWeightQuantizationConfig(bits=4, **model_compression_params),
|
||
**model_kwargs
|
||
)
|
||
ov_model.save_pretrained(int4_model_dir)
|
||
del ov_model
|
||
|
||
else:
|
||
convert_to_fp16()
|
||
ov_model = ov.Core().read_model(fp16_model_dir / "openvino_model.xml")
|
||
shutil.copy(fp16_model_dir / "config.json", int4_model_dir / "config.json")
|
||
configuration_file = fp16_model_dir / f"configuration_{model_type}.py"
|
||
if configuration_file.exists():
|
||
shutil.copy(
|
||
configuration_file, int4_model_dir / f"configuration_{model_type}.py"
|
||
)
|
||
mode = nncf.CompressWeightsMode.INT4_SYM if model_compression_params["sym"] else \
|
||
nncf.CompressWeightsMode.INT4_ASYM
|
||
del model_compression_params["sym"]
|
||
compressed_model = nncf.compress_weights(ov_model, mode=mode, **model_compression_params)
|
||
ov.save_model(compressed_model, int4_model_dir / "openvino_model.xml")
|
||
del ov_model
|
||
del compressed_model
|
||
gc.collect()
|
||
|
||
|
||
if prepare_fp16_model.value:
|
||
convert_to_fp16()
|
||
if prepare_int8_model.value:
|
||
convert_to_int8()
|
||
if prepare_int4_model.value:
|
||
convert_to_int4()
|
||
|
||
Let’s compare model size for different compression types
|
||
|
||
.. code:: ipython3
|
||
|
||
fp16_weights = fp16_model_dir / "openvino_model.bin"
|
||
int8_weights = int8_model_dir / "openvino_model.bin"
|
||
int4_weights = int4_model_dir / "openvino_model.bin"
|
||
|
||
if fp16_weights.exists():
|
||
print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
|
||
for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
|
||
if compressed_weights.exists():
|
||
print(
|
||
f"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB"
|
||
)
|
||
if compressed_weights.exists() and fp16_weights.exists():
|
||
print(
|
||
f"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}"
|
||
)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Size of model with INT4 compressed weights is 1837.58 MB
|
||
|
||
|
||
Convert embedding model
|
||
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
Since some embedding models can only support limited languages, we can
|
||
filter them out according the LLM you selected.
|
||
|
||
.. code:: ipython3
|
||
|
||
embedding_model_id = list(SUPPORTED_EMBEDDING_MODELS)
|
||
|
||
if model_language.value != "Chinese":
|
||
embedding_model_id = [x for x in embedding_model_id if "chinese" not in x]
|
||
|
||
embedding_model_id = widgets.Dropdown(
|
||
options=embedding_model_id,
|
||
value=embedding_model_id[0],
|
||
description="Embedding Model:",
|
||
disabled=False,
|
||
)
|
||
|
||
embedding_model_id
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Embedding Model:', options=('all-mpnet-base-v2',), value='all-mpnet-base-v2')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
embedding_model_configuration = SUPPORTED_EMBEDDING_MODELS[embedding_model_id.value]
|
||
print(f"Selected {embedding_model_id.value} model")
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Selected all-mpnet-base-v2 model
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
embedding_model_dir = Path(embedding_model_id.value)
|
||
|
||
if not (embedding_model_dir / "openvino_model.xml").exists():
|
||
model = AutoModel.from_pretrained(embedding_model_configuration["model_id"])
|
||
converters[embedding_model_id.value](model, embedding_model_dir)
|
||
tokenizer = AutoTokenizer.from_pretrained(embedding_model_configuration["model_id"])
|
||
tokenizer.save_pretrained(embedding_model_dir)
|
||
del model
|
||
|
||
Select device for inference and model variant
|
||
---------------------------------------------
|
||
|
||
|
||
|
||
**Note**: There may be no speedup for INT4/INT8 compressed models on
|
||
dGPU.
|
||
|
||
Select device for embedding model inference
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
core = ov.Core()
|
||
embedding_device = widgets.Dropdown(
|
||
options=core.available_devices + ["AUTO"],
|
||
value="CPU",
|
||
description="Device:",
|
||
disabled=False,
|
||
)
|
||
|
||
embedding_device
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
print(f"Embedding model will be loaded to {embedding_device.value} device for response generation")
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Embedding model will be loaded to CPU device for response generation
|
||
|
||
|
||
Select device for LLM model inference
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
llm_device = widgets.Dropdown(
|
||
options=core.available_devices + ["AUTO"],
|
||
value="CPU",
|
||
description="Device:",
|
||
disabled=False,
|
||
)
|
||
|
||
llm_device
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
print(f"LLM model will be loaded to {llm_device.value} device for response generation")
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
LLM model will be loaded to CPU device for response generation
|
||
|
||
|
||
Load model
|
||
----------
|
||
|
||
|
||
|
||
Load embedding model
|
||
~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
Wrapper around a text embedding model for LangChain, used for converting
|
||
text to embeddings.
|
||
|
||
.. code:: ipython3
|
||
|
||
from ov_embedding_model import OVEmbeddings
|
||
|
||
embedding = OVEmbeddings.from_model_id(
|
||
embedding_model_dir,
|
||
do_norm=embedding_model_configuration["do_norm"],
|
||
ov_config={
|
||
"device_name": embedding_device.value,
|
||
"config": {"PERFORMANCE_HINT": "THROUGHPUT"},
|
||
},
|
||
model_kwargs={
|
||
"model_max_length": 512,
|
||
},
|
||
)
|
||
|
||
Load LLM model
|
||
~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
The cell below create ``OVMPTModel`` and ``OVCHATGLM2Model`` wrapper
|
||
based on ``OVModelForCausalLM`` model.
|
||
|
||
.. code:: ipython3
|
||
|
||
from ov_llm_model import model_classes
|
||
|
||
.. code:: ipython3
|
||
|
||
available_models = []
|
||
if int4_model_dir.exists():
|
||
available_models.append("INT4")
|
||
if int8_model_dir.exists():
|
||
available_models.append("INT8")
|
||
if fp16_model_dir.exists():
|
||
available_models.append("FP16")
|
||
|
||
model_to_run = widgets.Dropdown(
|
||
options=available_models,
|
||
value=available_models[0],
|
||
description="Model to run:",
|
||
disabled=False,
|
||
)
|
||
|
||
model_to_run
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Model to run:', options=('FP16',), value='FP16')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
from langchain.llms import HuggingFacePipeline
|
||
|
||
if model_to_run.value == "INT4":
|
||
model_dir = int4_model_dir
|
||
elif model_to_run.value == "INT8":
|
||
model_dir = int8_model_dir
|
||
else:
|
||
model_dir = fp16_model_dir
|
||
print(f"Loading model from {model_dir}")
|
||
|
||
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
|
||
|
||
# On a GPU device a model is executed in FP16 precision. For red-pajama-3b-chat model there known accuracy
|
||
# issues caused by this, which we avoid by setting precision hint to "f32".
|
||
if llm_model_id.value == "red-pajama-3b-chat" and "GPU" in core.available_devices and llm_device.value in ["GPU", "AUTO"]:
|
||
ov_config["INFERENCE_PRECISION_HINT"] = "f32"
|
||
|
||
model_name = llm_model_configuration["model_id"]
|
||
stop_tokens = llm_model_configuration.get("stop_tokens")
|
||
class_key = llm_model_id.value.split("-")[0]
|
||
tok = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
||
|
||
class StopOnTokens(StoppingCriteria):
|
||
def __init__(self, token_ids):
|
||
self.token_ids = token_ids
|
||
|
||
def __call__(
|
||
self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs
|
||
) -> bool:
|
||
for stop_id in self.token_ids:
|
||
if input_ids[0][-1] == stop_id:
|
||
return True
|
||
return False
|
||
|
||
if stop_tokens is not None:
|
||
if isinstance(stop_tokens[0], str):
|
||
stop_tokens = tok.convert_tokens_to_ids(stop_tokens)
|
||
|
||
stop_tokens = [StopOnTokens(stop_tokens)]
|
||
|
||
model_class = (
|
||
OVModelForCausalLM
|
||
if not llm_model_configuration["remote"]
|
||
else model_classes[class_key]
|
||
)
|
||
ov_model = model_class.from_pretrained(
|
||
model_dir,
|
||
device=llm_device.value,
|
||
ov_config=ov_config,
|
||
config=AutoConfig.from_pretrained(model_dir, trust_remote_code=True),
|
||
trust_remote_code=True,
|
||
)
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Loading model from chatglm3-6b/FP16
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
tokenizer_config.json: 0%| | 0.00/1.41k [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
tokenization_chatglm.py: 0%| | 0.00/13.0k [00:00<?, ?B/s]
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
A new version of the following files was downloaded from https://huggingface.co/THUDM/chatglm3-6b:
|
||
- tokenization_chatglm.py
|
||
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
tokenizer.model: 0%| | 0.00/1.02M [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
special_tokens_map.json: 0%| | 0.00/3.00 [00:00<?, ?B/s]
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Setting eos_token is not supported, use the default one.
|
||
Setting pad_token is not supported, use the default one.
|
||
Setting unk_token is not supported, use the default one.
|
||
The argument `trust_remote_code` is to be used along with export=True. It will be ignored.
|
||
Compiling the model to CPU ...
|
||
|
||
|
||
Run QA over Document
|
||
--------------------
|
||
|
||
|
||
|
||
Now, when model created, we can setup Chatbot interface using
|
||
`Gradio <https://www.gradio.app/>`__.
|
||
|
||
A typical RAG application has two main components:
|
||
|
||
- **Indexing**: a pipeline for ingesting data from a source and
|
||
indexing it. This usually happen offline.
|
||
|
||
- **Retrieval and generation**: the actual RAG chain, which takes the
|
||
user query at run time and retrieves the relevant data from the
|
||
index, then passes that to the model.
|
||
|
||
The most common full sequence from raw data to answer looks like:
|
||
|
||
**Indexing** 1. ``Load``: First we need to load our data. We’ll use
|
||
DocumentLoaders for this. 2. ``Split``: Text splitters break large
|
||
Documents into smaller chunks. This is useful both for indexing data and
|
||
for passing it in to a model, since large chunks are harder to search
|
||
over and won’t in a model’s finite context window. 3. ``Store``: We need
|
||
somewhere to store and index our splits, so that they can later be
|
||
searched over. This is often done using a VectorStore and Embeddings
|
||
model.
|
||
|
||
.. figure:: https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/dfed2ba3-0c3a-4e0e-a2a7-01638730486a
|
||
:alt: Indexing pipeline
|
||
|
||
Indexing pipeline
|
||
|
||
**Retrieval and generation** 1. ``Retrieve``: Given a user input,
|
||
relevant splits are retrieved from storage using a Retriever. 2.
|
||
``Generate``: A LLM produces an answer using a prompt that includes the
|
||
question and the retrieved data.
|
||
|
||
.. figure:: https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/f0545ddc-c0cd-4569-8c86-9879fdab105a
|
||
:alt: Retrieval and generation pipeline
|
||
|
||
Retrieval and generation pipeline
|
||
|
||
.. code:: ipython3
|
||
|
||
from typing import List
|
||
from langchain.text_splitter import CharacterTextSplitter, RecursiveCharacterTextSplitter, MarkdownTextSplitter
|
||
from langchain.document_loaders import (
|
||
CSVLoader,
|
||
EverNoteLoader,
|
||
PDFMinerLoader,
|
||
TextLoader,
|
||
UnstructuredEPubLoader,
|
||
UnstructuredHTMLLoader,
|
||
UnstructuredMarkdownLoader,
|
||
UnstructuredODTLoader,
|
||
UnstructuredPowerPointLoader,
|
||
UnstructuredWordDocumentLoader, )
|
||
|
||
|
||
class ChineseTextSplitter(CharacterTextSplitter):
|
||
def __init__(self, pdf: bool = False, **kwargs):
|
||
super().__init__(**kwargs)
|
||
self.pdf = pdf
|
||
|
||
def split_text(self, text: str) -> List[str]:
|
||
if self.pdf:
|
||
text = re.sub(r"\n{3,}", "\n", text)
|
||
text = text.replace("\n\n", "")
|
||
sent_sep_pattern = re.compile(
|
||
'([﹒﹔﹖﹗．。！？]["’”」』]{0,2}|(?=["‘“「『]{1,2}|$))')
|
||
sent_list = []
|
||
for ele in sent_sep_pattern.split(text):
|
||
if sent_sep_pattern.match(ele) and sent_list:
|
||
sent_list[-1] += ele
|
||
elif ele:
|
||
sent_list.append(ele)
|
||
return sent_list
|
||
|
||
|
||
TEXT_SPLITERS = {
|
||
"Character": CharacterTextSplitter,
|
||
"RecursiveCharacter": RecursiveCharacterTextSplitter,
|
||
"Markdown": MarkdownTextSplitter,
|
||
"Chinese": ChineseTextSplitter,
|
||
}
|
||
|
||
|
||
LOADERS = {
|
||
".csv": (CSVLoader, {}),
|
||
".doc": (UnstructuredWordDocumentLoader, {}),
|
||
".docx": (UnstructuredWordDocumentLoader, {}),
|
||
".enex": (EverNoteLoader, {}),
|
||
".epub": (UnstructuredEPubLoader, {}),
|
||
".html": (UnstructuredHTMLLoader, {}),
|
||
".md": (UnstructuredMarkdownLoader, {}),
|
||
".odt": (UnstructuredODTLoader, {}),
|
||
".pdf": (PDFMinerLoader, {}),
|
||
".ppt": (UnstructuredPowerPointLoader, {}),
|
||
".pptx": (UnstructuredPowerPointLoader, {}),
|
||
".txt": (TextLoader, {"encoding": "utf8"}),
|
||
}
|
||
|
||
To reuse RAG pipeline of LangChain, we can wrapper around a LLM/chat
|
||
model for LangChain, used for generating the response text. An OpenVINO
|
||
compiled model can be run locally through the ``HuggingFacePipeline``
|
||
class.
|
||
|
||
.. code:: ipython3
|
||
|
||
from langchain.prompts import PromptTemplate
|
||
from langchain.vectorstores import Chroma
|
||
from langchain.chains import RetrievalQA
|
||
from langchain.docstore.document import Document
|
||
from threading import Event, Thread
|
||
import gradio as gr
|
||
import re
|
||
from uuid import uuid4
|
||
|
||
|
||
def load_single_document(file_path: str) -> List[Document]:
|
||
"""
|
||
helper for loading a single document
|
||
|
||
Params:
|
||
file_path: document path
|
||
Returns:
|
||
documents loaded
|
||
|
||
"""
|
||
ext = "." + file_path.rsplit(".", 1)[-1]
|
||
if ext in LOADERS:
|
||
loader_class, loader_args = LOADERS[ext]
|
||
loader = loader_class(file_path, **loader_args)
|
||
return loader.load()
|
||
|
||
raise ValueError(f"File does not exist '{ext}'")
|
||
|
||
|
||
def default_partial_text_processor(partial_text: str, new_text: str):
|
||
"""
|
||
helper for updating partially generated answer, used by default
|
||
|
||
Params:
|
||
partial_text: text buffer for storing previosly generated text
|
||
new_text: text update for the current step
|
||
Returns:
|
||
updated text string
|
||
|
||
"""
|
||
partial_text += new_text
|
||
return partial_text
|
||
|
||
|
||
text_processor = llm_model_configuration.get(
|
||
"partial_text_processor", default_partial_text_processor
|
||
)
|
||
|
||
|
||
def build_retriever(docs, spliter_name, chunk_size, chunk_overlap, vector_search_top_k):
|
||
"""
|
||
Initialize a vector database
|
||
|
||
Params:
|
||
doc: orignal documents provided by user
|
||
chunk_size: size of a single sentence chunk
|
||
chunk_overlap: overlap size between 2 chunks
|
||
vector_search_top_k: Vector search top k
|
||
|
||
"""
|
||
documents = []
|
||
for doc in docs:
|
||
documents.extend(load_single_document(doc.name))
|
||
|
||
text_splitter = TEXT_SPLITERS[spliter_name](
|
||
chunk_size=chunk_size, chunk_overlap=chunk_overlap
|
||
)
|
||
|
||
texts = text_splitter.split_documents(documents)
|
||
|
||
db = Chroma.from_documents(texts, embedding)
|
||
|
||
global retriever
|
||
retriever = db.as_retriever(search_kwargs={"k": vector_search_top_k})
|
||
|
||
return "Retriever is Ready"
|
||
|
||
|
||
def user(message, history):
|
||
"""
|
||
callback function for updating user messages in interface on submit button click
|
||
|
||
Params:
|
||
message: current message
|
||
history: conversation history
|
||
Returns:
|
||
None
|
||
"""
|
||
# Append the user's message to the conversation history
|
||
return "", history + [[message, ""]]
|
||
|
||
|
||
def bot(history, temperature, top_p, top_k, repetition_penalty, conversation_id):
|
||
"""
|
||
callback function for running chatbot on submit button click
|
||
|
||
Params:
|
||
history: conversation history
|
||
temperature: parameter for control the level of creativity in AI-generated text.
|
||
By adjusting the `temperature`, you can influence the AI model's probability distribution, making the text more focused or diverse.
|
||
top_p: parameter for control the range of tokens considered by the AI model based on their cumulative probability.
|
||
top_k: parameter for control the range of tokens considered by the AI model based on their cumulative probability, selecting number of tokens with highest probability.
|
||
repetition_penalty: parameter for penalizing tokens based on how frequently they occur in the text.
|
||
conversation_id: unique conversation identifier.
|
||
|
||
"""
|
||
streamer = TextIteratorStreamer(
|
||
tok, timeout=60.0, skip_prompt=True, skip_special_tokens=True
|
||
)
|
||
generate_kwargs = dict(
|
||
model=ov_model,
|
||
tokenizer=tok,
|
||
max_new_tokens=256,
|
||
temperature=temperature,
|
||
do_sample=temperature > 0.0,
|
||
top_p=top_p,
|
||
top_k=top_k,
|
||
repetition_penalty=repetition_penalty,
|
||
streamer=streamer,
|
||
)
|
||
if stop_tokens is not None:
|
||
generate_kwargs["stopping_criteria"] = StoppingCriteriaList(stop_tokens)
|
||
|
||
pipe = pipeline("text-generation", **generate_kwargs)
|
||
llm = HuggingFacePipeline(pipeline=pipe)
|
||
|
||
prompt = PromptTemplate.from_template(llm_model_configuration["rag_prompt_template"])
|
||
chain_type_kwargs = {"prompt": prompt}
|
||
rag_chain = RetrievalQA.from_chain_type(
|
||
llm=llm,
|
||
chain_type="stuff",
|
||
retriever=retriever,
|
||
chain_type_kwargs=chain_type_kwargs,
|
||
)
|
||
|
||
stream_complete = Event()
|
||
|
||
def infer(question):
|
||
rag_chain.invoke(question)
|
||
stream_complete.set()
|
||
|
||
t1 = Thread(target=infer, args=(history[-1][0],))
|
||
t1.start()
|
||
|
||
# Initialize an empty string to store the generated text
|
||
partial_text = ""
|
||
for new_text in streamer:
|
||
partial_text = text_processor(partial_text, new_text)
|
||
history[-1][1] = partial_text
|
||
yield history
|
||
|
||
|
||
def request_cancel():
|
||
ov_model.request.cancel()
|
||
|
||
|
||
def get_uuid():
|
||
"""
|
||
universal unique identifier for thread
|
||
"""
|
||
return str(uuid4())
|
||
|
||
|
||
with gr.Blocks(
|
||
theme=gr.themes.Soft(),
|
||
css=".disclaimer {font-variant-caps: all-small-caps;}",
|
||
) as demo:
|
||
conversation_id = gr.State(get_uuid)
|
||
gr.Markdown("""<h1><center>QA over Document</center></h1>""")
|
||
gr.Markdown(f"""<center>Powered by OpenVINO and {llm_model_id.value} </center>""")
|
||
with gr.Row():
|
||
with gr.Column(scale=1):
|
||
docs = gr.File(
|
||
label="Load text files",
|
||
file_count="multiple",
|
||
file_types=[
|
||
".csv",
|
||
".doc",
|
||
".docx",
|
||
".enex",
|
||
".epub",
|
||
".html",
|
||
".md",
|
||
".odt",
|
||
".pdf",
|
||
".ppt",
|
||
".pptx",
|
||
".txt",
|
||
],
|
||
)
|
||
load_docs = gr.Button("Build Retriever")
|
||
retriever_argument = gr.Accordion("Retriever Configuration", open=False)
|
||
with retriever_argument:
|
||
spliter = gr.Dropdown(
|
||
["Character", "RecursiveCharacter", "Markdown", "Chinese"],
|
||
value="RecursiveCharacter",
|
||
label="Text Spliter",
|
||
info="Method used to splite the documents",
|
||
multiselect=False,
|
||
)
|
||
|
||
chunk_size = gr.Slider(
|
||
label="Chunk size",
|
||
value=1000,
|
||
minimum=100,
|
||
maximum=2000,
|
||
step=50,
|
||
interactive=True,
|
||
info="Size of sentence chunk",
|
||
)
|
||
|
||
chunk_overlap = gr.Slider(
|
||
label="Chunk overlap",
|
||
value=200,
|
||
minimum=0,
|
||
maximum=400,
|
||
step=10,
|
||
interactive=True,
|
||
info=("Overlap between 2 chunks"),
|
||
)
|
||
|
||
vector_search_top_k = gr.Slider(
|
||
1,
|
||
10,
|
||
value=4,
|
||
step=1,
|
||
label="Vector search top k",
|
||
interactive=True,
|
||
)
|
||
langchain_status = gr.Textbox(
|
||
label="Status", value="Retriever is Not ready", interactive=False
|
||
)
|
||
with gr.Accordion("Generation Configuration", open=False):
|
||
with gr.Row():
|
||
with gr.Column():
|
||
with gr.Row():
|
||
temperature = gr.Slider(
|
||
label="Temperature",
|
||
value=0.1,
|
||
minimum=0.0,
|
||
maximum=1.0,
|
||
step=0.1,
|
||
interactive=True,
|
||
info="Higher values produce more diverse outputs",
|
||
)
|
||
with gr.Column():
|
||
with gr.Row():
|
||
top_p = gr.Slider(
|
||
label="Top-p (nucleus sampling)",
|
||
value=1.0,
|
||
minimum=0.0,
|
||
maximum=1,
|
||
step=0.01,
|
||
interactive=True,
|
||
info=(
|
||
"Sample from the smallest possible set of tokens whose cumulative probability "
|
||
"exceeds top_p. Set to 1 to disable and sample from all tokens."
|
||
),
|
||
)
|
||
with gr.Column():
|
||
with gr.Row():
|
||
top_k = gr.Slider(
|
||
label="Top-k",
|
||
value=50,
|
||
minimum=0.0,
|
||
maximum=200,
|
||
step=1,
|
||
interactive=True,
|
||
info="Sample from a shortlist of top-k tokens — 0 to disable and sample from all tokens.",
|
||
)
|
||
with gr.Column():
|
||
with gr.Row():
|
||
repetition_penalty = gr.Slider(
|
||
label="Repetition Penalty",
|
||
value=1.1,
|
||
minimum=1.0,
|
||
maximum=2.0,
|
||
step=0.1,
|
||
interactive=True,
|
||
info="Penalize repetition — 1.0 to disable.",
|
||
)
|
||
with gr.Column(scale=4):
|
||
chatbot = gr.Chatbot(height=600)
|
||
with gr.Row():
|
||
with gr.Column():
|
||
msg = gr.Textbox(
|
||
label="Chat Message Box",
|
||
placeholder="Chat Message Box",
|
||
show_label=False,
|
||
container=False,
|
||
)
|
||
with gr.Column():
|
||
with gr.Row():
|
||
submit = gr.Button("Submit")
|
||
stop = gr.Button("Stop")
|
||
clear = gr.Button("Clear")
|
||
load_docs.click(
|
||
build_retriever,
|
||
inputs=[docs, spliter, chunk_size, chunk_overlap, vector_search_top_k],
|
||
outputs=[langchain_status],
|
||
queue=False,
|
||
)
|
||
submit_event = msg.submit(
|
||
user, [msg, chatbot], [msg, chatbot], queue=False
|
||
).then(bot, [chatbot, temperature, top_p, top_k, repetition_penalty, conversation_id], chatbot, queue=True)
|
||
submit_click_event = submit.click(
|
||
user, [msg, chatbot], [msg, chatbot], queue=False
|
||
).then(bot, [chatbot, temperature, top_p, top_k, repetition_penalty, conversation_id], chatbot, queue=True)
|
||
stop.click(
|
||
fn=request_cancel,
|
||
inputs=None,
|
||
outputs=None,
|
||
cancels=[submit_event, submit_click_event],
|
||
queue=False,
|
||
)
|
||
clear.click(lambda: None, None, chatbot, queue=False)
|
||
|
||
demo.queue()
|
||
# if you are launching remotely, specify server_name and server_port
|
||
# demo.launch(server_name='your server name', server_port='server port in int')
|
||
# if you have any issue to launch on your platform, you can pass share=True to launch method:
|
||
# demo.launch(share=True)
|
||
# it creates a publicly shareable link for the interface. Read more in the docs: https://gradio.app/docs/
|
||
demo.launch()
|
||
|
||
.. code:: ipython3
|
||
|
||
# please run this cell for stopping gradio interface
|
||
demo.close()
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Closing server running on port: 5579
|
||
|