natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] Updating Tutorials (#24435) 

Updating `Interactive Tutorials` section.
This commit is contained in:
Sebastian Golebiewski 2024-05-10 08:32:53 +02:00 committed by GitHub
parent d203f80943
commit 3549ff32ea
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
259 changed files with 8914 additions and 71974 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/nbdoc/consts.py
View File

@ -6,7 +6,7 @@ repo_directory = "notebooks"
repo_owner = "openvinotoolkit"
repo_name = "openvino_notebooks"
repo_branch = "tree/main"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20240417220808/dist/rst_files/"
artifacts_link = "http://repository.toolbox.iotg.sclab.intel.com/projects/ov-notebook/0.1.0-latest/20240506220807/dist/rst_files/"
blacklisted_extensions = ['.xml', '.bin']
notebooks_repo = "https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/"
notebooks_binder = "https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath="

955
docs/notebooks/3D-pose-estimation-with-output.rst
View File

File diff suppressed because it is too large Load Diff

8
docs/notebooks/3D-segmentation-point-clouds-with-output.rst
View File

@ -48,10 +48,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -220,7 +216,7 @@ chair for example.
.. parsed-literal::
/tmp/ipykernel_3630/2434168836.py:12: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
/tmp/ipykernel_2840820/2434168836.py:12: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
ax.scatter3D(X, Y, Z, s=5, cmap="jet", marker="o", label="chair")
@ -321,7 +317,7 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
/tmp/ipykernel_3630/2804603389.py:23: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
/tmp/ipykernel_2840820/2804603389.py:23: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
ax.scatter(XCur, YCur, ZCur, s=5, cmap="jet", marker="o", label=classes[i])

4
docs/notebooks/action-recognition-webcam-with-output_files/action-recognition-webcam-with-output_22_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3a88055dc2dbfefd2c65a09518b7c3d76951fee1961547834059ed7dadb633cc
size 69011
oid sha256:d81ad2656ac29b01ccc926bde66d6d66d05d187c5bf95b2767317a6355ad86fb
size 68213

26
docs/notebooks/all_notebooks_paths.txt
View File

@ -4,35 +4,25 @@ notebooks/action-recognition-webcam/action-recognition-webcam.ipynb
notebooks/amused-lightweight-text-to-image/amused-lightweight-text-to-image.ipynb
notebooks/animate-anyone/animate-anyone.ipynb
notebooks/async-api/async-api.ipynb
notebooks/attention-center/attention-center.ipynb
notebooks/auto-device/auto-device.ipynb
notebooks/bark-text-to-audio/bark-text-to-audio.ipynb
notebooks/big-transfer-quantization/tensorflow-bit-image-classification-nncf-quantization.ipynb
notebooks/blip-diffusion-subject-generation/blip-diffusion-subject-generation.ipynb
notebooks/blip-visual-language-processing/blip-visual-language-processing.ipynb
notebooks/clip-language-saliency-map/clip-language-saliency-map.ipynb
notebooks/clip-zero-shot-image-classification/clip-zero-shot-classification.ipynb
notebooks/code-language-id/code-language-id.ipynb
notebooks/controlnet-stable-diffusion/controlnet-stable-diffusion.ipynb
notebooks/convert-to-openvino/convert-to-openvino.ipynb
notebooks/convert-to-openvino/legacy-mo-convert-to-openvino.ipynb
notebooks/cross-lingual-books-alignment/cross-lingual-books-alignment.ipynb
notebooks/ct-segmentation-quantize/ct-scan-live-inference.ipynb
notebooks/ct-segmentation-quantize/ct-segmentation-quantize-nncf.ipynb
notebooks/ct-segmentation-quantize/data-preparation-ct-scan.ipynb
notebooks/ct-segmentation-quantize/pytorch-monai-training.ipynb
notebooks/decidiffusion-image-generation/decidiffusion-image-generation.ipynb
notebooks/deepfloyd-if/deep-floyd-if-convert.ipynb
notebooks/deepfloyd-if/deep-floyd-if-optimize.ipynb
notebooks/depth-anything/depth-anything.ipynb
notebooks/depth-estimation-videpth/depth-estimation-videpth.ipynb
notebooks/detectron2-to-openvino/detectron2-to-openvino.ipynb
notebooks/distilbert-sequence-classification/distilbert-sequence-classification.ipynb
notebooks/distil-whisper-asr/distil-whisper-asr.ipynb
notebooks/dolly-2-instruction-following/dolly-2-instruction-following.ipynb
notebooks/efficient-sam/efficient-sam.ipynb
notebooks/encodec-audio-compression/encodec-audio-compression.ipynb
notebooks/fastcomposer-image-generation/fastcomposer-image-generation.ipynb
notebooks/fast-segment-anything/fast-segment-anything.ipynb
notebooks/film-slowmo/film-slowmo.ipynb
notebooks/freevc-voice-conversion/freevc-voice-conversion.ipynb
@ -41,6 +31,7 @@ notebooks/grammar-correction/grammar-correction.ipynb
notebooks/grounded-segment-anything/grounded-segment-anything.ipynb
notebooks/handwritten-ocr/handwritten-ocr.ipynb
notebooks/hello-detection/hello-detection.ipynb
notebooks/hello-npu/hello-npu.ipynb
notebooks/hello-segmentation/hello-segmentation.ipynb
notebooks/hello-world/hello-world.ipynb
notebooks/hugging-face-hub/hugging-face-hub.ipynb
@ -62,7 +53,6 @@ notebooks/llm-agent-langchain/llm-agent-langchain.ipynb
notebooks/llm-chatbot/llm-chatbot.ipynb
notebooks/llm-question-answering/llm-question-answering.ipynb
notebooks/llm-rag-langchain/llm-rag-langchain.ipynb
notebooks/machine-translation/machine-translation.ipynb
notebooks/magika-content-type-recognition/magika-content-type-recognition.ipynb
notebooks/meter-reader/meter-reader.ipynb
notebooks/mms-massively-multilingual-speech/mms-massively-multilingual-speech.ipynb
@ -82,8 +72,6 @@ notebooks/optimize-preprocessing/optimize-preprocessing.ipynb
notebooks/paddle-ocr-webcam/paddle-ocr-webcam.ipynb
notebooks/paddle-to-openvino/paddle-to-openvino-classification.ipynb
notebooks/paint-by-example/paint-by-example.ipynb
notebooks/performance-tricks/latency-tricks.ipynb
notebooks/performance-tricks/throughput-tricks.ipynb
notebooks/person-tracking-webcam/person-tracking.ipynb
notebooks/photo-maker/photo-maker.ipynb
notebooks/pix2struct-docvqa/pix2struct-docvqa.ipynb
@ -96,14 +84,12 @@ notebooks/pytorch-to-openvino/pytorch-to-openvino.ipynb
notebooks/qrcode-monster/qrcode-monster.ipynb
notebooks/quantizing-model-with-accuracy-control/speech-recognition-quantization-wav2vec2.ipynb
notebooks/quantizing-model-with-accuracy-control/yolov8-quantization-with-accuracy-control.ipynb
notebooks/question-answering/question-answering.ipynb
notebooks/riffusion-text-to-music/riffusion-text-to-music.ipynb
notebooks/rmbg-background-removal/rmbg-background-removal.ipynb
notebooks/sdxl-turbo/sdxl-turbo.ipynb
notebooks/segment-anything/segment-anything.ipynb
notebooks/segmenter-semantic-segmentation/segmenter-semantic-segmentation.ipynb
notebooks/siglip-zero-shot-image-classification/siglip-zero-shot-image-classification.ipynb
notebooks/slowfast-video-recognition/slowfast-video-recognition.ipynb
notebooks/sketch-to-image-pix2pix-turbo/sketch-to-image-pix2pix-turbo.ipynb
notebooks/softvc-voice-conversion/softvc-voice-conversion.ipynb
notebooks/sound-generation-audioldm2/sound-generation-audioldm2.ipynb
notebooks/sparsity-optimization/sparsity-optimization.ipynb
@ -111,6 +97,7 @@ notebooks/speculative-sampling/speculative-sampling.ipynb
notebooks/speech-recognition-quantization/speech-recognition-quantization-data2vec.ipynb
notebooks/speech-recognition-quantization/speech-recognition-quantization-wav2vec2.ipynb
notebooks/speech-to-text/speech-to-text.ipynb
notebooks/stable-cascade-image-generation/stable-cascade-image-generation.ipynb
notebooks/stable-diffusion-ip-adapter/stable-diffusion-ip-adapter.ipynb
notebooks/stable-diffusion-keras-cv/stable-diffusion-keras-cv.ipynb
notebooks/stable-diffusion-text-to-image/stable-diffusion-text-to-image.ipynb
@ -120,8 +107,8 @@ notebooks/stable-diffusion-v2/stable-diffusion-v2-optimum-demo.ipynb
notebooks/stable-diffusion-v2/stable-diffusion-v2-text-to-image-demo.ipynb
notebooks/stable-diffusion-v2/stable-diffusion-v2-text-to-image.ipynb
notebooks/stable-diffusion-xl/segmind-vegart.ipynb
notebooks/stable-diffusion-xl/ssd-b1.ipynb
notebooks/stable-diffusion-xl/stable-diffusion-xl.ipynb
notebooks/stable-video-diffusion/stable-video-diffusion.ipynb
notebooks/stable-zephyr-3b-chatbot/stable-zephyr-3b-chatbot.ipynb
notebooks/style-transfer-webcam/style-transfer.ipynb
notebooks/surya-line-level-text-detection/surya-line-level-text-detection.ipynb
@ -131,14 +118,11 @@ notebooks/tensorflow-hub/tensorflow-hub.ipynb
notebooks/tensorflow-object-detection-to-openvino/tensorflow-instance-segmentation-to-openvino.ipynb
notebooks/tensorflow-object-detection-to-openvino/tensorflow-object-detection-to-openvino.ipynb
notebooks/tensorflow-quantization-aware-training/tensorflow-quantization-aware-training.ipynb
notebooks/tensorflow-training-openvino/tensorflow-training-openvino.ipynb
notebooks/tensorflow-training-openvino/tensorflow-training-openvino-nncf.ipynb
notebooks/text-prediction/text-prediction.ipynb
notebooks/tflite-selfie-segmentation/tflite-selfie-segmentation.ipynb
notebooks/tflite-to-openvino/tflite-to-openvino.ipynb
notebooks/tiny-sd-image-generation/tiny-sd-image-generation.ipynb
notebooks/torchvision-zoo-to-openvino/convnext-classification.ipynb
notebooks/torchvision-zoo-to-openvino/lraspp-segmentation.ipynb
notebooks/triposr-3d-reconstruction/triposr-3d-reconstruction.ipynb
notebooks/typo-detector/typo-detector.ipynb
notebooks/vehicle-detection-and-recognition/vehicle-detection-and-recognition.ipynb
@ -147,8 +131,6 @@ notebooks/vision-image-colorization/vision-image-colorization.ipynb
notebooks/vision-monodepth/vision-monodepth.ipynb
notebooks/vision-paddlegan-anime/vision-paddlegan-anime.ipynb
notebooks/vision-paddlegan-superresolution/vision-paddlegan-superresolution.ipynb
notebooks/vision-superresolution/vision-superresolution-image.ipynb
notebooks/vision-superresolution/vision-superresolution-video.ipynb
notebooks/whisper-subtitles-generation/whisper-convert.ipynb
notebooks/whisper-subtitles-generation/whisper-nncf-quantize.ipynb
notebooks/wuerstchen-image-generation/wuerstchen-image-generation.ipynb

441
docs/notebooks/amused-lightweight-text-to-image-with-output.rst
View File

@ -50,18 +50,9 @@ Prerequisites
%pip install -q "nncf>=2.9.0" datasets
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -74,17 +65,23 @@ Load and run the original pipeline
import torch
from diffusers import AmusedPipeline
pipe = AmusedPipeline.from_pretrained(
"amused/amused-256",
)
prompt = "kind smiling ghost"
image = pipe(prompt, generator=torch.Generator("cpu").manual_seed(8)).images[0]
image.save("text2image_256.png")
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
.. parsed-literal::
@ -133,8 +130,8 @@ Define paths for converted models:
.. code:: ipython3
from pathlib import Path
TRANSFORMER_OV_PATH = Path("models/transformer_ir.xml")
TEXT_ENCODER_OV_PATH = Path("models/text_encoder_ir.xml")
VQVAE_OV_PATH = Path("models/vqvae_ir.xml")
@ -147,10 +144,10 @@ file.
.. code:: ipython3
import torch
import openvino as ov
def convert(model: torch.nn.Module, xml_path: str, example_input):
xml_path = Path(xml_path)
if not xml_path.exists():
@ -158,7 +155,7 @@ file.
with torch.no_grad():
converted_model = ov.convert_model(model, example_input=example_input)
ov.save_model(converted_model, xml_path, compress_to_fp16=False)
# cleanup memory
torch._C._jit_clear_class_registry()
torch.jit._recursive.concrete_type_store = torch.jit._recursive.ConcreteTypeStore()
@ -175,17 +172,17 @@ Convert the Text Encoder
def __init__(self, text_encoder):
super().__init__()
self.text_encoder = text_encoder
def forward(self, input_ids=None, return_dict=None, output_hidden_states=None):
outputs = self.text_encoder(
input_ids=input_ids,
return_dict=return_dict,
output_hidden_states=output_hidden_states,
)
return outputs.text_embeds, outputs.last_hidden_state, outputs.hidden_states
input_ids = pipe.tokenizer(
prompt,
return_tensors="pt",
@ -193,45 +190,41 @@ Convert the Text Encoder
truncation=True,
max_length=pipe.tokenizer.model_max_length,
)
input_example = {
"input_ids": input_ids.input_ids,
"return_dict": torch.tensor(True),
"output_hidden_states": torch.tensor(True),
}
convert(TextEncoderWrapper(pipe.text_encoder), TEXT_ENCODER_OV_PATH, input_example)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:86: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:86: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if input_shape[-1] > 1 or self.sliding_window is not None:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:162: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:162: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if past_key_values_length > 0:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:622: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:620: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
encoder_states = () if output_hidden_states else None
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:627: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:625: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if output_hidden_states:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:281: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:279: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:289: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:321: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:319: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:650: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:648: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if output_hidden_states:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:653: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:651: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if not return_dict:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:744: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:742: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if not return_dict:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:1229: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:1227: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if not return_dict:
@ -246,7 +239,7 @@ Convert the U-ViT transformer
def __init__(self, transformer):
super().__init__()
self.transformer = transformer
def forward(
self,
latents=None,
@ -260,21 +253,21 @@ Convert the U-ViT transformer
pooled_text_emb=pooled_text_emb,
encoder_hidden_states=encoder_hidden_states,
)
shape = (1, 16, 16)
latents = torch.full(shape, pipe.scheduler.config.mask_token_id, dtype=torch.long)
latents = torch.cat([latents] * 2)
example_input = {
"latents": latents,
"micro_conds": torch.rand([2, 5], dtype=torch.float32),
"pooled_text_emb": torch.rand([2, 768], dtype=torch.float32),
"encoder_hidden_states": torch.rand([2, 77, 768], dtype=torch.float32),
}
pipe.transformer.eval()
w_transformer = TransformerWrapper(pipe.transformer)
convert(w_transformer, TRANSFORMER_OV_PATH, example_input)
@ -293,7 +286,7 @@ suitable. This function repeats part of ``AmusedPipeline``.
shape = (1, 16, 16)
latents = torch.full(shape, pipe.scheduler.config.mask_token_id, dtype=torch.long)
model_input = torch.cat([latents] * 2)
model_output = pipe.transformer(
model_input,
micro_conds=torch.rand([2, 5], dtype=torch.float32),
@ -303,50 +296,50 @@ suitable. This function repeats part of ``AmusedPipeline``.
guidance_scale = 10.0
uncond_logits, cond_logits = model_output.chunk(2)
model_output = uncond_logits + guidance_scale * (cond_logits - uncond_logits)
latents = pipe.scheduler.step(
model_output=model_output,
timestep=torch.tensor(0),
sample=latents,
).prev_sample
return latents
class VQVAEWrapper(torch.nn.Module):
def __init__(self, vqvae):
super().__init__()
self.vqvae = vqvae
def forward(self, latents=None, force_not_quantize=True, shape=None):
outputs = self.vqvae.decode(
latents,
force_not_quantize=force_not_quantize,
shape=shape.tolist(),
)
return outputs
latents = get_latents()
example_vqvae_input = {
"latents": latents,
"force_not_quantize": torch.tensor(True),
"shape": torch.tensor((1, 16, 16, 64)),
}
convert(VQVAEWrapper(pipe.vqvae), VQVAE_OV_PATH, example_vqvae_input)
.. parsed-literal::
/tmp/ipykernel_4430/3779428577.py:34: TracerWarning: Converting a tensor to a Python list might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/tmp/ipykernel_2841622/3779428577.py:34: TracerWarning: Converting a tensor to a Python list might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
shape=shape.tolist(),
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/vq_model.py:144: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/vq_model.py:144: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if not force_not_quantize:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert hidden_states.shape[1] == self.channels
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if hidden_states.shape[0] >= 64:
@ -360,8 +353,8 @@ Select device from dropdown list for running inference using OpenVINO.
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
@ -369,7 +362,7 @@ Select device from dropdown list for running inference using OpenVINO.
description="Device:",
disabled=False,
)
device
@ -394,29 +387,29 @@ wrapper classes return ``torch.Tensor``\ s instead of ``np.array``\ s.
.. code:: ipython3
from collections import namedtuple
class ConvTextEncoderWrapper(torch.nn.Module):
def __init__(self, text_encoder, config):
super().__init__()
self.config = config
self.text_encoder = text_encoder
def forward(self, input_ids=None, return_dict=None, output_hidden_states=None):
inputs = {
"input_ids": input_ids,
"return_dict": return_dict,
"output_hidden_states": output_hidden_states,
}
outs = self.text_encoder(inputs)
outputs = namedtuple("CLIPTextModelOutput", ("text_embeds", "last_hidden_state", "hidden_states"))
text_embeds = torch.from_numpy(outs[0])
last_hidden_state = torch.from_numpy(outs[1])
hidden_states = list(torch.from_numpy(out) for out in outs.values())[2:]
return outputs(text_embeds, last_hidden_state, hidden_states)
.. code:: ipython3
@ -426,7 +419,7 @@ wrapper classes return ``torch.Tensor``\ s instead of ``np.array``\ s.
super().__init__()
self.config = config
self.transformer = transformer
def forward(self, latents=None, micro_conds=None, pooled_text_emb=None, encoder_hidden_states=None, **kwargs):
outputs = self.transformer(
{
@ -437,7 +430,7 @@ wrapper classes return ``torch.Tensor``\ s instead of ``np.array``\ s.
},
share_inputs=False,
)
return torch.from_numpy(outputs[0])
.. code:: ipython3
@ -448,17 +441,17 @@ wrapper classes return ``torch.Tensor``\ s instead of ``np.array``\ s.
self.vqvae = vqvae
self.dtype = dtype
self.config = config
def decode(self, latents=None, force_not_quantize=True, shape=None):
inputs = {
"latents": latents,
"force_not_quantize": force_not_quantize,
"shape": torch.tensor(shape),
}
outs = self.vqvae(inputs)
outs = namedtuple("VQVAE", "sample")(torch.from_numpy(outs[0]))
return outs
And insert wrappers instances in the pipeline:
@ -466,25 +459,25 @@ And insert wrappers instances in the pipeline:
.. code:: ipython3
prompt = "kind smiling ghost"
transformer = pipe.transformer
vqvae = pipe.vqvae
text_encoder = pipe.text_encoder
pipe.__dict__["_internal_dict"]["_execution_device"] = pipe._execution_device # this is to avoid some problem that can occur in the pipeline
pipe.register_modules(
text_encoder=ConvTextEncoderWrapper(ov_text_encoder, text_encoder.config),
transformer=ConvTransformerWrapper(ov_transformer, transformer.config),
vqvae=ConvVQVAEWrapper(ov_vqvae, vqvae.dtype, vqvae.config),
)
image = pipe(prompt, generator=torch.Generator("cpu").manual_seed(8)).images[0]
image.save("text2image_256.png")
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
@ -547,13 +540,13 @@ improve model inference speed.
.. code:: ipython3
QUANTIZED_TRANSFORMER_OV_PATH = Path(str(TRANSFORMER_OV_PATH).replace(".xml", "_quantized.xml"))
to_quantize = widgets.Checkbox(
value=False,
value=True,
description="Quantization",
disabled=False,
)
to_quantize
@ -561,19 +554,19 @@ improve model inference speed.
.. parsed-literal::
Checkbox(value=False, description='Quantization')
Checkbox(value=True, description='Quantization')
.. code:: ipython3
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
)
open("skip_kernel_extension.py", "w").write(r.text)
%load_ext skip_kernel_extension
Prepare calibration dataset
@ -589,42 +582,42 @@ model inputs for calibration we customize ``CompiledModel``.
.. code:: ipython3
%%skip not $to_quantize.value
import datasets
from tqdm.auto import tqdm
from typing import Any, Dict, List
import pickle
import numpy as np
def disable_progress_bar(pipeline, disable=True):
if not hasattr(pipeline, "_progress_bar_config"):
pipeline._progress_bar_config = {'disable': disable}
else:
pipeline._progress_bar_config['disable'] = disable
class CompiledModelDecorator(ov.CompiledModel):
def __init__(self, compiled_model: ov.CompiledModel, data_cache: List[Any] = None, keep_prob: float = 0.5):
super().__init__(compiled_model)
self.data_cache = data_cache if data_cache is not None else []
self.keep_prob = keep_prob
def __call__(self, *args, **kwargs):
if np.random.rand() <= self.keep_prob:
self.data_cache.append(*args)
return super().__call__(*args, **kwargs)
def collect_calibration_data(ov_transformer_model, calibration_dataset_size: int) -> List[Dict]:
calibration_dataset_filepath = Path(f"calibration_data/{calibration_dataset_size}.pkl")
if not calibration_dataset_filepath.exists():
calibration_data = []
pipe.transformer.transformer = CompiledModelDecorator(ov_transformer_model, calibration_data, keep_prob=1.0)
disable_progress_bar(pipe)
dataset = datasets.load_dataset("conceptual_captions", split="train").shuffle(seed=42)
# Run inference for data collection
pbar = tqdm(total=calibration_dataset_size)
for batch in dataset:
@ -635,14 +628,14 @@ model inputs for calibration we customize ``CompiledModel``.
pbar.update(len(calibration_data) - pbar.n)
if pbar.n >= calibration_dataset_size:
break
pipe.transformer.transformer = ov_transformer_model
disable_progress_bar(pipe, disable=False)
calibration_dataset_filepath.parent.mkdir(exist_ok=True, parents=True)
with open(calibration_dataset_filepath, 'wb') as f:
pickle.dump(calibration_data, f)
with open(calibration_dataset_filepath, 'rb') as f:
calibration_data = pickle.load(f)
return calibration_data
@ -658,12 +651,14 @@ model.
.. code:: ipython3
%%skip not $to_quantize.value
from nncf.quantization.advanced_parameters import AdvancedSmoothQuantParameters
from nncf.quantization.range_estimator import RangeEstimatorParameters, StatisticsCollectorParameters, StatisticsType, \
AggregatorType
import nncf
CALIBRATION_DATASET_SIZE = 12 * 200
CALIBRATION_DATASET_SIZE = 12 * 25
if not QUANTIZED_TRANSFORMER_OV_PATH.exists():
calibration_data = collect_calibration_data(ov_transformer, CALIBRATION_DATASET_SIZE)
quantized_model = nncf.quantize(
@ -674,27 +669,177 @@ model.
# We ignore convolutions to improve quality of generations without significant drop in inference speed
ignored_scope=nncf.IgnoredScope(types=["Convolution"]),
# Value of 0.85 was obtained using grid search based on Inception Score computed below
advanced_parameters=nncf.AdvancedQuantizationParameters(smooth_quant_alphas=AdvancedSmoothQuantParameters(matmul=0.85))
advanced_parameters=nncf.AdvancedQuantizationParameters(
smooth_quant_alphas=AdvancedSmoothQuantParameters(matmul=0.85),
# During activation statistics collection we ignore 1% of outliers which improves quantization quality
activations_range_estimator_params=RangeEstimatorParameters(
min=StatisticsCollectorParameters(statistics_type=StatisticsType.MIN,
aggregator_type=AggregatorType.MEAN_NO_OUTLIERS,
quantile_outlier_prob=0.01),
max=StatisticsCollectorParameters(statistics_type=StatisticsType.MAX,
aggregator_type=AggregatorType.MEAN_NO_OUTLIERS,
quantile_outlier_prob=0.01)
)
)
)
ov.save_model(quantized_model, QUANTIZED_TRANSFORMER_OV_PATH)
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1486: FutureWarning: The repository for conceptual_captions contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/conceptual_captions
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
warnings.warn(
.. parsed-literal::
0%| | 0/300 [00:00<?, ?it/s]
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
.. parsed-literal::
Output()
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
Output()
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
INFO:nncf:3 ignored nodes were found by types in the NNCFGraph
INFO:nncf:182 ignored nodes were found by name in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 37 __module.transformer.embed.conv/aten::_convolution/Convolution
INFO:nncf:Not adding activation input quantizer for operation: 2883 __module.transformer.mlm_layer.conv1/aten::_convolution/Convolution
INFO:nncf:Not adding activation input quantizer for operation: 3243 __module.transformer.mlm_layer.conv2/aten::_convolution/Convolution
.. parsed-literal::
Output()
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
Demo generation with quantized pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%skip not $to_quantize.value
original_ov_transformer_model = pipe.transformer.transformer
pipe.transformer.transformer = core.compile_model(QUANTIZED_TRANSFORMER_OV_PATH, device.value)
image = pipe(prompt, generator=torch.Generator('cpu').manual_seed(8)).images[0]
image.save('text2image_256_quantized.png')
pipe.transformer.transformer = original_ov_transformer_model
display(image)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
.. parsed-literal::
0%| | 0/12 [00:00<?, ?it/s]
.. image:: amused-lightweight-text-to-image-with-output_files/amused-lightweight-text-to-image-with-output_37_2.png
Compute Inception Scores and inference time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -712,24 +857,24 @@ a rough estimate of generation quality.
.. code:: ipython3
%%skip not $to_quantize.value
from torchmetrics.image.inception import InceptionScore
from torchvision import transforms as transforms
from itertools import islice
import time
VALIDATION_DATASET_SIZE = 100
def compute_inception_score(ov_transformer_model_path, validation_set_size, batch_size=100):
original_ov_transformer_model = pipe.transformer.transformer
pipe.transformer.transformer = core.compile_model(ov_transformer_model_path, device.value)
disable_progress_bar(pipe)
dataset = datasets.load_dataset("conceptual_captions", "unlabeled", split="validation").shuffle(seed=42)
dataset = islice(dataset, validation_set_size)
inception_score = InceptionScore(normalize=True, splits=1)
images = []
infer_times = []
for batch in tqdm(dataset, total=validation_set_size, desc="Computing Inception Score"):
@ -741,27 +886,65 @@ a rough estimate of generation quality.
infer_times.append(time.perf_counter() - start_time)
image = transforms.ToTensor()(image)
images.append(image)
mean_perf_time = sum(infer_times) / len(infer_times)
while len(images) > 0:
images_batch = torch.stack(images[-batch_size:])
images = images[:-batch_size]
inception_score.update(images_batch)
kl_mean, kl_std = inception_score.compute()
pipe.transformer.transformer = original_ov_transformer_model
disable_progress_bar(pipe, disable=False)
return kl_mean, mean_perf_time
original_inception_score, original_time = compute_inception_score(TRANSFORMER_OV_PATH, VALIDATION_DATASET_SIZE)
print(f"Original pipeline Inception Score: {original_inception_score}")
quantized_inception_score, quantized_time = compute_inception_score(QUANTIZED_TRANSFORMER_OV_PATH, VALIDATION_DATASET_SIZE)
print(f"Quantized pipeline Inception Score: {quantized_inception_score}")
print(f"Quantization speed-up: {original_time / quantized_time:.2f}x")
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: Metric `InceptionScore` will save all extracted features in buffer. For large datasets this may lead to large memory footprint.
warnings.warn(\*args, \*\*kwargs) # noqa: B028
.. parsed-literal::
Computing Inception Score: 0%| | 0/100 [00:00<?, ?it/s]
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/image/inception.py:176: UserWarning: std(): degrees of freedom is <= 0. Correction should be strictly less than the reduction factor (input numel divided by output numel). (Triggered internally at ../aten/src/ATen/native/ReduceOps.cpp:1807.)
return kl.mean(), kl.std()
.. parsed-literal::
Original pipeline Inception Score: 11.146076202392578
.. parsed-literal::
Computing Inception Score: 0%| | 0/100 [00:00<?, ?it/s]
.. parsed-literal::
Quantized pipeline Inception Score: 9.630990028381348
Quantization speed-up: 2.10x
Interactive inference
---------------------
@ -772,13 +955,13 @@ Below you can select which pipeline to run: original or quantized.
.. code:: ipython3
quantized_model_present = QUANTIZED_TRANSFORMER_OV_PATH.exists()
use_quantized_model = widgets.Checkbox(
value=True if quantized_model_present else False,
description="Use quantized pipeline",
disabled=not quantized_model_present,
)
use_quantized_model
@ -786,7 +969,7 @@ Below you can select which pipeline to run: original or quantized.
.. parsed-literal::
Checkbox(value=False, description='Use quantized pipeline', disabled=True)
Checkbox(value=True, description='Use quantized pipeline')
@ -794,18 +977,18 @@ Below you can select which pipeline to run: original or quantized.
import gradio as gr
import numpy as np
pipe.transformer.transformer = core.compile_model(
QUANTIZED_TRANSFORMER_OV_PATH if use_quantized_model.value else TRANSFORMER_OV_PATH,
device.value,
)
def generate(prompt, seed, _=gr.Progress(track_tqdm=True)):
image = pipe(prompt, generator=torch.Generator("cpu").manual_seed(seed)).images[0]
return image
demo = gr.Interface(
generate,
[
@ -832,7 +1015,7 @@ Below you can select which pipeline to run: original or quantized.
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.

3
docs/notebooks/amused-lightweight-text-to-image-with-output_files/amused-lightweight-text-to-image-with-output_37_2.jpg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7f3abc24fb5cb6c674974109aeb1e98eb87dfa188eaf36ffd314e0ed5d370824
size 5343

3
docs/notebooks/amused-lightweight-text-to-image-with-output_files/amused-lightweight-text-to-image-with-output_37_2.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c76a4f5e4a5d2f65626843461cf1b637efdef7c8da3cba9ff63baf9602845e70
size 78975

177
docs/notebooks/animate-anyone-with-output.rst
View File

File diff suppressed because one or more lines are too long

791
docs/notebooks/async-api-with-output.rst
View File

@ -1,791 +0,0 @@
Asynchronous Inference with OpenVINO™
=====================================
This notebook demonstrates how to use the `Async
API <https://docs.openvino.ai/2024/openvino-workflow/running-inference/optimize-inference/general-optimizations.html>`__
for asynchronous execution with OpenVINO.
OpenVINO Runtime supports inference in either synchronous or
asynchronous mode. The key advantage of the Async API is that when a
device is busy with inference, the application can perform other tasks
in parallel (for example, populating inputs or scheduling other
requests) rather than wait for the current inference to complete first.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Prepare model and data
processing <#prepare-model-and-data-processing>`__
- `Download test model <#download-test-model>`__
- `Load the model <#load-the-model>`__
- `Create functions for data
processing <#create-functions-for-data-processing>`__
- `Get the test video <#get-the-test-video>`__
- `How to improve the throughput of video
processing <#how-to-improve-the-throughput-of-video-processing>`__
- `Sync Mode (default) <#sync-mode-default>`__
- `Test performance in Sync Mode <#test-performance-in-sync-mode>`__
- `Async Mode <#async-mode>`__
- `Test the performance in Async
Mode <#test-the-performance-in-async-mode>`__
- `Compare the performance <#compare-the-performance>`__
- `AsyncInferQueue <#asyncinferqueue>`__
- `Setting Callback <#setting-callback>`__
- `Test the performance with
AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__
Imports
-------
.. code:: ipython3
import platform
%pip install -q "openvino>=2023.1.0"
%pip install -q opencv-python
if platform.system() != "windows":
%pip install -q "matplotlib>=3.4"
else:
%pip install -q "matplotlib>=3.4,<3.7"
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. code:: ipython3
import cv2
import time
import numpy as np
import openvino as ov
from IPython import display
import matplotlib.pyplot as plt
# Fetch the notebook utils script from the openvino_notebooks repo
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
import notebook_utils as utils
Prepare model and data processing
---------------------------------
Download test model
~~~~~~~~~~~~~~~~~~~
We use a pre-trained model from OpenVINO’s `Open Model
Zoo <https://docs.openvino.ai/2024/documentation/legacy-features/model-zoo.html>`__
to start the test. In this case, the model will be executed to detect
the person in each frame of the video.
.. code:: ipython3
# directory where model will be downloaded
base_model_dir = "model"
# model name as named in Open Model Zoo
model_name = "person-detection-0202"
precision = "FP16"
model_path = f"model/intel/{model_name}/{precision}/{model_name}.xml"
download_command = f"omz_downloader " f"--name {model_name} " f"--precision {precision} " f"--output_dir {base_model_dir} " f"--cache_dir {base_model_dir}"
! $download_command
.. parsed-literal::
################|| Downloading person-detection-0202 ||################
========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.xml
.. parsed-literal::
... 12%, 32 KB, 998 KB/s, 0 seconds passed
.. parsed-literal::
... 25%, 64 KB, 1010 KB/s, 0 seconds passed
... 38%, 96 KB, 1446 KB/s, 0 seconds passed
... 51%, 128 KB, 1336 KB/s, 0 seconds passed
... 64%, 160 KB, 1638 KB/s, 0 seconds passed
... 77%, 192 KB, 1927 KB/s, 0 seconds passed
... 89%, 224 KB, 2206 KB/s, 0 seconds passed
... 100%, 248 KB, 2423 KB/s, 0 seconds passed
========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.bin
.. parsed-literal::
... 0%, 32 KB, 646 KB/s, 0 seconds passed
.. parsed-literal::
... 1%, 64 KB, 991 KB/s, 0 seconds passed
... 2%, 96 KB, 1232 KB/s, 0 seconds passed
... 3%, 128 KB, 1330 KB/s, 0 seconds passed
... 4%, 160 KB, 1640 KB/s, 0 seconds passed
... 5%, 192 KB, 1860 KB/s, 0 seconds passed
.. parsed-literal::
... 6%, 224 KB, 2064 KB/s, 0 seconds passed
... 7%, 256 KB, 2271 KB/s, 0 seconds passed
... 8%, 288 KB, 2239 KB/s, 0 seconds passed
... 9%, 320 KB, 2476 KB/s, 0 seconds passed
... 9%, 352 KB, 2698 KB/s, 0 seconds passed
... 10%, 384 KB, 2906 KB/s, 0 seconds passed
... 11%, 416 KB, 3122 KB/s, 0 seconds passed
... 12%, 448 KB, 3335 KB/s, 0 seconds passed
... 13%, 480 KB, 3537 KB/s, 0 seconds passed
... 14%, 512 KB, 3723 KB/s, 0 seconds passed
... 15%, 544 KB, 3847 KB/s, 0 seconds passed
... 16%, 576 KB, 4016 KB/s, 0 seconds passed
.. parsed-literal::
... 17%, 608 KB, 3756 KB/s, 0 seconds passed
... 18%, 640 KB, 3938 KB/s, 0 seconds passed
... 18%, 672 KB, 4121 KB/s, 0 seconds passed
... 19%, 704 KB, 4300 KB/s, 0 seconds passed
... 20%, 736 KB, 4484 KB/s, 0 seconds passed
... 21%, 768 KB, 4666 KB/s, 0 seconds passed
... 22%, 800 KB, 4751 KB/s, 0 seconds passed
... 23%, 832 KB, 4819 KB/s, 0 seconds passed
... 24%, 864 KB, 4944 KB/s, 0 seconds passed
... 25%, 896 KB, 5048 KB/s, 0 seconds passed
... 26%, 928 KB, 5115 KB/s, 0 seconds passed
... 27%, 960 KB, 5202 KB/s, 0 seconds passed
... 27%, 992 KB, 5285 KB/s, 0 seconds passed
... 28%, 1024 KB, 5369 KB/s, 0 seconds passed
... 29%, 1056 KB, 5452 KB/s, 0 seconds passed
... 30%, 1088 KB, 5568 KB/s, 0 seconds passed
... 31%, 1120 KB, 5674 KB/s, 0 seconds passed
... 32%, 1152 KB, 5773 KB/s, 0 seconds passed
... 33%, 1184 KB, 5877 KB/s, 0 seconds passed
... 34%, 1216 KB, 5951 KB/s, 0 seconds passed
... 35%, 1248 KB, 6041 KB/s, 0 seconds passed
.. parsed-literal::
... 36%, 1280 KB, 6132 KB/s, 0 seconds passed
... 36%, 1312 KB, 6225 KB/s, 0 seconds passed
... 37%, 1344 KB, 6354 KB/s, 0 seconds passed
... 38%, 1376 KB, 6446 KB/s, 0 seconds passed
... 39%, 1408 KB, 6522 KB/s, 0 seconds passed
... 40%, 1440 KB, 6650 KB/s, 0 seconds passed
... 41%, 1472 KB, 6739 KB/s, 0 seconds passed
... 42%, 1504 KB, 6827 KB/s, 0 seconds passed
... 43%, 1536 KB, 6935 KB/s, 0 seconds passed
... 44%, 1568 KB, 7017 KB/s, 0 seconds passed
... 45%, 1600 KB, 7124 KB/s, 0 seconds passed
... 45%, 1632 KB, 7201 KB/s, 0 seconds passed
... 46%, 1664 KB, 7306 KB/s, 0 seconds passed
... 47%, 1696 KB, 7411 KB/s, 0 seconds passed
... 48%, 1728 KB, 7526 KB/s, 0 seconds passed
... 49%, 1760 KB, 7611 KB/s, 0 seconds passed
... 50%, 1792 KB, 7697 KB/s, 0 seconds passed
... 51%, 1824 KB, 7816 KB/s, 0 seconds passed
... 52%, 1856 KB, 7912 KB/s, 0 seconds passed
... 53%, 1888 KB, 8016 KB/s, 0 seconds passed
... 54%, 1920 KB, 8115 KB/s, 0 seconds passed
... 54%, 1952 KB, 8215 KB/s, 0 seconds passed
... 55%, 1984 KB, 8313 KB/s, 0 seconds passed
... 56%, 2016 KB, 8409 KB/s, 0 seconds passed
... 57%, 2048 KB, 8507 KB/s, 0 seconds passed
... 58%, 2080 KB, 8608 KB/s, 0 seconds passed
... 59%, 2112 KB, 8706 KB/s, 0 seconds passed
... 60%, 2144 KB, 8801 KB/s, 0 seconds passed
... 61%, 2176 KB, 8893 KB/s, 0 seconds passed
... 62%, 2208 KB, 8987 KB/s, 0 seconds passed
... 63%, 2240 KB, 9085 KB/s, 0 seconds passed
... 64%, 2272 KB, 9202 KB/s, 0 seconds passed
... 64%, 2304 KB, 9302 KB/s, 0 seconds passed
... 65%, 2336 KB, 9396 KB/s, 0 seconds passed
... 66%, 2368 KB, 9489 KB/s, 0 seconds passed
... 67%, 2400 KB, 9579 KB/s, 0 seconds passed
... 68%, 2432 KB, 9694 KB/s, 0 seconds passed
... 69%, 2464 KB, 9791 KB/s, 0 seconds passed
... 70%, 2496 KB, 9879 KB/s, 0 seconds passed
... 71%, 2528 KB, 9993 KB/s, 0 seconds passed
... 72%, 2560 KB, 10088 KB/s, 0 seconds passed
... 73%, 2592 KB, 10186 KB/s, 0 seconds passed
... 73%, 2624 KB, 10299 KB/s, 0 seconds passed
... 74%, 2656 KB, 10376 KB/s, 0 seconds passed
... 75%, 2688 KB, 10472 KB/s, 0 seconds passed
... 76%, 2720 KB, 10566 KB/s, 0 seconds passed
... 77%, 2752 KB, 10673 KB/s, 0 seconds passed
.. parsed-literal::
... 78%, 2784 KB, 10764 KB/s, 0 seconds passed
... 79%, 2816 KB, 10874 KB/s, 0 seconds passed
... 80%, 2848 KB, 10964 KB/s, 0 seconds passed
... 81%, 2880 KB, 11062 KB/s, 0 seconds passed
... 82%, 2912 KB, 11172 KB/s, 0 seconds passed
... 82%, 2944 KB, 11241 KB/s, 0 seconds passed
... 83%, 2976 KB, 11346 KB/s, 0 seconds passed
... 84%, 3008 KB, 11451 KB/s, 0 seconds passed
... 85%, 3040 KB, 11528 KB/s, 0 seconds passed
... 86%, 3072 KB, 11634 KB/s, 0 seconds passed
... 87%, 3104 KB, 11720 KB/s, 0 seconds passed
... 88%, 3136 KB, 11817 KB/s, 0 seconds passed
... 89%, 3168 KB, 11921 KB/s, 0 seconds passed
... 90%, 3200 KB, 12009 KB/s, 0 seconds passed
... 91%, 3232 KB, 12114 KB/s, 0 seconds passed
... 91%, 3264 KB, 12197 KB/s, 0 seconds passed
... 92%, 3296 KB, 12274 KB/s, 0 seconds passed
... 93%, 3328 KB, 12377 KB/s, 0 seconds passed
... 94%, 3360 KB, 12462 KB/s, 0 seconds passed
... 95%, 3392 KB, 12563 KB/s, 0 seconds passed
... 96%, 3424 KB, 12652 KB/s, 0 seconds passed
... 97%, 3456 KB, 12756 KB/s, 0 seconds passed
... 98%, 3488 KB, 12845 KB/s, 0 seconds passed
... 99%, 3520 KB, 12947 KB/s, 0 seconds passed
... 100%, 3549 KB, 13037 KB/s, 0 seconds passed
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
import ipywidgets as widgets
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="CPU",
description="Device:",
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
Load the model
~~~~~~~~~~~~~~
.. code:: ipython3
# initialize OpenVINO runtime
core = ov.Core()
# read the network and corresponding weights from file
model = core.read_model(model=model_path)
# compile the model for the CPU (you can choose manually CPU, GPU etc.)
# or let the engine choose the best available device (AUTO)
compiled_model = core.compile_model(model=model, device_name=device.value)
# get input node
input_layer_ir = model.input(0)
N, C, H, W = input_layer_ir.shape
shape = (H, W)
Create functions for data processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
def preprocess(image):
"""
Define the preprocess function for input data
:param: image: the orignal input frame
:returns:
resized_image: the image processed
"""
resized_image = cv2.resize(image, shape)
resized_image = cv2.cvtColor(np.array(resized_image), cv2.COLOR_BGR2RGB)
resized_image = resized_image.transpose((2, 0, 1))
resized_image = np.expand_dims(resized_image, axis=0).astype(np.float32)
return resized_image
def postprocess(result, image, fps):
"""
Define the postprocess function for output data
:param: result: the inference results
image: the orignal input frame
fps: average throughput calculated for each frame
:returns:
image: the image with bounding box and fps message
"""
detections = result.reshape(-1, 7)
for i, detection in enumerate(detections):
_, image_id, confidence, xmin, ymin, xmax, ymax = detection
if confidence > 0.5:
xmin = int(max((xmin * image.shape[1]), 10))
ymin = int(max((ymin * image.shape[0]), 10))
xmax = int(min((xmax * image.shape[1]), image.shape[1] - 10))
ymax = int(min((ymax * image.shape[0]), image.shape[0] - 10))
cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (0, 255, 0), 2)
cv2.putText(
image,
str(round(fps, 2)) + " fps",
(5, 20),
cv2.FONT_HERSHEY_SIMPLEX,
0.7,
(0, 255, 0),
3,
)
return image
Get the test video
~~~~~~~~~~~~~~~~~~
.. code:: ipython3
video_path = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/CEO%20Pat%20Gelsinger%20on%20Leading%20Intel.mp4"
How to improve the throughput of video processing
-------------------------------------------------
Below, we compare the performance of the synchronous and async-based
approaches:
Sync Mode (default)
~~~~~~~~~~~~~~~~~~~
Let us see how video processing works with the default approach. Using
the synchronous approach, the frame is captured with OpenCV and then
immediately processed:
.. figure:: https://user-images.githubusercontent.com/91237924/168452573-d354ea5b-7966-44e5-813d-f9053be4338a.png
:alt: drawing
drawing
::
while(true) {
// capture frame
// populate CURRENT InferRequest
// Infer CURRENT InferRequest
//this call is synchronous
// display CURRENT result
}
\``\`
.. code:: ipython3
def sync_api(source, flip, fps, use_popup, skip_first_frames):
"""
Define the main function for video processing in sync mode
:param: source: the video path or the ID of your webcam
:returns:
sync_fps: the inference throughput in sync mode
"""
frame_number = 0
infer_request = compiled_model.create_infer_request()
player = None
try:
# Create a video player
player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
# Start capturing
start_time = time.time()
player.start()
if use_popup:
title = "Press ESC to Exit"
cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)
while True:
frame = player.next()
if frame is None:
print("Source ended")
break
resized_frame = preprocess(frame)
infer_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
# Start the inference request in synchronous mode
infer_request.infer()
res = infer_request.get_output_tensor(0).data
stop_time = time.time()
total_time = stop_time - start_time
frame_number = frame_number + 1
sync_fps = frame_number / total_time
frame = postprocess(res, frame, sync_fps)
# Display the results
if use_popup:
cv2.imshow(title, frame)
key = cv2.waitKey(1)
# escape = 27
if key == 27:
break
else:
# Encode numpy array to jpg
_, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
# Create IPython image
i = display.Image(data=encoded_img)
# Display the image in this notebook
display.clear_output(wait=True)
display.display(i)
# ctrl-c
except KeyboardInterrupt:
print("Interrupted")
# Any different error
except RuntimeError as e:
print(e)
finally:
if use_popup:
cv2.destroyAllWindows()
if player is not None:
# stop capturing
player.stop()
return sync_fps
Test performance in Sync Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
sync_fps = sync_api(source=video_path, flip=False, fps=30, use_popup=False, skip_first_frames=800)
print(f"average throuput in sync mode: {sync_fps:.2f} fps")
.. image:: async-api-with-output_files/async-api-with-output_17_0.png
.. parsed-literal::
Source ended
average throuput in sync mode: 44.01 fps
Async Mode
~~~~~~~~~~
Let us see how the OpenVINO Async API can improve the overall frame rate
of an application. The key advantage of the Async approach is as
follows: while a device is busy with the inference, the application can
do other things in parallel (for example, populating inputs or
scheduling other requests) rather than wait for the current inference to
complete first.
.. figure:: https://user-images.githubusercontent.com/91237924/168452572-c2ff1c59-d470-4b85-b1f6-b6e1dac9540e.png
:alt: drawing
drawing
In the example below, inference is applied to the results of the video
decoding. So it is possible to keep multiple infer requests, and while
the current request is processed, the input frame for the next is being
captured. This essentially hides the latency of capturing, so that the
overall frame rate is rather determined only by the slowest part of the
pipeline (decoding vs inference) and not by the sum of the stages.
::
while(true) {
// capture frame
// populate NEXT InferRequest
// start NEXT InferRequest
// this call is async and returns immediately
// wait for the CURRENT InferRequest
// display CURRENT result
// swap CURRENT and NEXT InferRequests
}
.. code:: ipython3
def async_api(source, flip, fps, use_popup, skip_first_frames):
"""
Define the main function for video processing in async mode
:param: source: the video path or the ID of your webcam
:returns:
async_fps: the inference throughput in async mode
"""
frame_number = 0
# Create 2 infer requests
curr_request = compiled_model.create_infer_request()
next_request = compiled_model.create_infer_request()
player = None
async_fps = 0
try:
# Create a video player
player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
# Start capturing
start_time = time.time()
player.start()
if use_popup:
title = "Press ESC to Exit"
cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)
# Capture CURRENT frame
frame = player.next()
resized_frame = preprocess(frame)
curr_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
# Start the CURRENT inference request
curr_request.start_async()
while True:
# Capture NEXT frame
next_frame = player.next()
if next_frame is None:
print("Source ended")
break
resized_frame = preprocess(next_frame)
next_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
# Start the NEXT inference request
next_request.start_async()
# Waiting for CURRENT inference result
curr_request.wait()
res = curr_request.get_output_tensor(0).data
stop_time = time.time()
total_time = stop_time - start_time
frame_number = frame_number + 1
async_fps = frame_number / total_time
frame = postprocess(res, frame, async_fps)
# Display the results
if use_popup:
cv2.imshow(title, frame)
key = cv2.waitKey(1)
# escape = 27
if key == 27:
break
else:
# Encode numpy array to jpg
_, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
# Create IPython image
i = display.Image(data=encoded_img)
# Display the image in this notebook
display.clear_output(wait=True)
display.display(i)
# Swap CURRENT and NEXT frames
frame = next_frame
# Swap CURRENT and NEXT infer requests
curr_request, next_request = next_request, curr_request
# ctrl-c
except KeyboardInterrupt:
print("Interrupted")
# Any different error
except RuntimeError as e:
print(e)
finally:
if use_popup:
cv2.destroyAllWindows()
if player is not None:
# stop capturing
player.stop()
return async_fps
Test the performance in Async Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
async_fps = async_api(source=video_path, flip=False, fps=30, use_popup=False, skip_first_frames=800)
print(f"average throuput in async mode: {async_fps:.2f} fps")
.. image:: async-api-with-output_files/async-api-with-output_21_0.png
.. parsed-literal::
Source ended
average throuput in async mode: 73.74 fps
Compare the performance
~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
width = 0.4
fontsize = 14
plt.rc("font", size=fontsize)
fig, ax = plt.subplots(1, 1, figsize=(10, 8))
rects1 = ax.bar([0], sync_fps, width, color="#557f2d")
rects2 = ax.bar([width], async_fps, width)
ax.set_ylabel("frames per second")
ax.set_xticks([0, width])
ax.set_xticklabels(["Sync mode", "Async mode"])
ax.set_xlabel("Higher is better")
fig.suptitle("Sync mode VS Async mode")
fig.tight_layout()
plt.show()
.. image:: async-api-with-output_files/async-api-with-output_23_0.png
``AsyncInferQueue``
-------------------
Asynchronous mode pipelines can be supported with the
`AsyncInferQueue <https://docs.openvino.ai/2024/openvino-workflow/running-inference/integrate-openvino-with-your-application/python-api-exclusives.html#asyncinferqueue>`__
wrapper class. This class automatically spawns the pool of
``InferRequest`` objects (also called “jobs”) and provides
synchronization mechanisms to control the flow of the pipeline. It is a
simpler way to manage the infer request queue in Asynchronous mode.
Setting Callback
~~~~~~~~~~~~~~~~
When ``callback`` is set, any job that ends inference calls upon the
Python function. The ``callback`` function must have two arguments: one
is the request that calls the ``callback``, which provides the
``InferRequest`` API; the other is called “user data”, which provides
the possibility of passing runtime values.
.. code:: ipython3
def callback(infer_request, info) -> None:
"""
Define the callback function for postprocessing
:param: infer_request: the infer_request object
info: a tuple includes original frame and starts time
:returns:
None
"""
global frame_number
global total_time
global inferqueue_fps
stop_time = time.time()
frame, start_time = info
total_time = stop_time - start_time
frame_number = frame_number + 1
inferqueue_fps = frame_number / total_time
res = infer_request.get_output_tensor(0).data[0]
frame = postprocess(res, frame, inferqueue_fps)
# Encode numpy array to jpg
_, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
# Create IPython image
i = display.Image(data=encoded_img)
# Display the image in this notebook
display.clear_output(wait=True)
display.display(i)
.. code:: ipython3
def inferqueue(source, flip, fps, skip_first_frames) -> None:
"""
Define the main function for video processing with async infer queue
:param: source: the video path or the ID of your webcam
:retuns:
None
"""
# Create infer requests queue
infer_queue = ov.AsyncInferQueue(compiled_model, 2)
infer_queue.set_callback(callback)
player = None
try:
# Create a video player
player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
# Start capturing
start_time = time.time()
player.start()
while True:
# Capture frame
frame = player.next()
if frame is None:
print("Source ended")
break
resized_frame = preprocess(frame)
# Start the inference request with async infer queue
infer_queue.start_async({input_layer_ir.any_name: resized_frame}, (frame, start_time))
except KeyboardInterrupt:
print("Interrupted")
# Any different error
except RuntimeError as e:
print(e)
finally:
infer_queue.wait_all()
player.stop()
Test the performance with ``AsyncInferQueue``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
frame_number = 0
total_time = 0
inferqueue(source=video_path, flip=False, fps=30, skip_first_frames=800)
print(f"average throughput in async mode with async infer queue: {inferqueue_fps:.2f} fps")
.. image:: async-api-with-output_files/async-api-with-output_29_0.png
.. parsed-literal::
average throughput in async mode with async infer queue: 112.89 fps

4
docs/notebooks/async-api-with-output_files/async-api-with-output_23_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3ee108fdee6cf1a56efa4a2347f0bc0c4cd09b6b30b121d114cfdd2cd87a9152
size 30443
oid sha256:deee8ff5a3fba807c811d63eeb7f516cbdc82d47f6b4c4ae797742ff6327a54d
size 30483

114
docs/notebooks/auto-device-with-output.rst
View File

@ -82,10 +82,6 @@ Import modules and create Core
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -190,15 +186,15 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
[23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:LOG_LEVEL=LOG_INFO
[23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
[23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
[23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERF_COUNT=NO
[23:04:35.7467]I[plugin.cpp:423][AUTO] device:CPU, priority:0
[23:04:35.7467]I[schedule.cpp:17][AUTO] scheduler starting
[23:04:35.7468]I[auto_schedule.cpp:131][AUTO] select device:CPU
[23:04:35.9061]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
[23:04:35.9063]I[plugin.cpp:451][AUTO] underlying hardware does not support hardware context
[23:25:36.8706]I[plugin.cpp:418][AUTO] device:CPU, config:LOG_LEVEL=LOG_INFO
[23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
[23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
[23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERF_COUNT=NO
[23:25:36.8707]I[plugin.cpp:423][AUTO] device:CPU, priority:0
[23:25:36.8707]I[schedule.cpp:17][AUTO] scheduler starting
[23:25:36.8707]I[auto_schedule.cpp:131][AUTO] select device:CPU
[23:25:37.0101]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
[23:25:37.0103]I[plugin.cpp:451][AUTO] underlying hardware does not support hardware context
Successfully compiled model without a device_name.
@ -212,7 +208,7 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
Deleted compiled_model
[23:04:35.9172]I[schedule.cpp:303][AUTO] scheduler ending
[23:25:37.0205]I[schedule.cpp:303][AUTO] scheduler ending
Explicitly pass AUTO as device_name to Core::compile_model API
@ -370,7 +366,7 @@ executed on CPU until GPU is ready.
.. parsed-literal::
Time to load model using AUTO device and get first inference: 0.15 seconds.
Time to load model using AUTO device and get first inference: 0.19 seconds.
.. code:: ipython3
@ -541,45 +537,13 @@ Loop for inference and update the FPS/Latency every
.. parsed-literal::
Compiling Model for AUTO device with THROUGHPUT hint
.. parsed-literal::
Start inference, 6 groups of FPS/latency will be measured over 10s intervals
.. parsed-literal::
throughput: 179.12fps, latency: 31.83ms, time interval: 10.00s
.. parsed-literal::
throughput: 181.09fps, latency: 32.33ms, time interval: 10.01s
.. parsed-literal::
throughput: 179.44fps, latency: 32.62ms, time interval: 10.00s
.. parsed-literal::
throughput: 179.98fps, latency: 32.57ms, time interval: 10.00s
.. parsed-literal::
throughput: 179.55fps, latency: 32.61ms, time interval: 10.01s
.. parsed-literal::
throughput: 179.60fps, latency: 32.58ms, time interval: 10.00s
.. parsed-literal::
throughput: 177.50fps, latency: 32.10ms, time interval: 10.00s
throughput: 179.46fps, latency: 32.64ms, time interval: 10.01s
throughput: 179.28fps, latency: 32.70ms, time interval: 10.00s
throughput: 177.92fps, latency: 32.86ms, time interval: 10.01s
throughput: 178.98fps, latency: 32.68ms, time interval: 10.02s
throughput: 178.91fps, latency: 32.77ms, time interval: 10.01s
Done
@ -624,45 +588,13 @@ Loop for inference and update the FPS/Latency for each
.. parsed-literal::
Compiling Model for AUTO Device with LATENCY hint
.. parsed-literal::
Start inference, 6 groups fps/latency will be out with 10s interval
.. parsed-literal::
throughput: 137.86fps, latency: 6.72ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.86fps, latency: 6.72ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.85fps, latency: 6.72ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.28fps, latency: 6.69ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.66fps, latency: 6.70ms, time interval: 10.00s
.. parsed-literal::
throughput: 140.48fps, latency: 6.68ms, time interval: 10.00s
.. parsed-literal::
throughput: 135.86fps, latency: 6.81ms, time interval: 10.00s
throughput: 138.93fps, latency: 6.82ms, time interval: 10.01s
throughput: 138.89fps, latency: 6.82ms, time interval: 10.00s
throughput: 138.82fps, latency: 6.81ms, time interval: 10.01s
throughput: 138.99fps, latency: 6.82ms, time interval: 10.00s
throughput: 139.01fps, latency: 6.82ms, time interval: 10.01s
Done

4
docs/notebooks/auto-device-with-output_files/auto-device-with-output_27_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84c73fa64ec907ad0be08db15bebc96b35c8b26b29d29f657f8ceeb75aab5662
size 27580
oid sha256:2ce598bcda980dc39d1fc49884a02509af4d0f599c4dd674e07b994af39cf533
size 27041

4
docs/notebooks/auto-device-with-output_files/auto-device-with-output_28_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61109ff91028e7d3caeed1fd125fd850785bb6341898e0dce5a87aece1166832
size 39983
oid sha256:1efc7ab0c2433842a374eb59dc1be264bb613039696dbdceb5b51c97c8ad666b
size 39972

1554
docs/notebooks/blip-diffusion-subject-generation-with-output.rst
View File

File diff suppressed because it is too large Load Diff

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/1c472f1f-1fce-4a13-9d44-b10f6f760ddb.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c52eadd32aeb484c638a81f268a10d121bc14b8500b7a5d8944f9104dd9dae1
size 111323

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_12_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a5a75d796a6a4ea9381e8fe6cbca907bdde7acb6417e8c34ab71adc712b6df38
size 495502

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_16_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d2c3acfc0cc530dbc8483f4e0fe39d954fbfca3ab1c682bd4e8b055822a9f1f5
size 680845

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_19_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6b1c442c8830c43c5c42c49236356c33997cf46a6f8dcd3a6226818c56f73f0a
size 541801

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_52_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:93593463b882f6dea67c4ee5173768b039b97a917eaf5bb8667b2bfb28c1cb8d
size 522726

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_55_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7e60816978a920b88168de568cc34b67a775da8790be93e723309953f5b0fc63
size 683108

3
docs/notebooks/blip-diffusion-subject-generation-with-output_files/blip-diffusion-subject-generation-with-output_58_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b954b7ef6368e8d354a06354f7822d438aa62e7a644e1f7a87f0f79035ec06e
size 539707

15
docs/notebooks/blip-visual-language-processing-with-output.rst
View File

@ -323,6 +323,10 @@ text and vision modalities and postprocessing of generation results.
.. code:: ipython3
from pathlib import Path
if not Path("./utils.py").exists():
download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/blip-visual-language-processing/utils.py")
from utils import visualize_results
fig = visualize_results(raw_image, answer, question)
@ -380,7 +384,6 @@ shape, containing RGB image pixel values normalized in the [0,1] range.
# if openvino model does not exist, convert it to IR
if not VISION_MODEL_OV.exists():
# export pytorch model to ov.Model
with torch.no_grad():
ov_vision_model = ov.convert_model(vision_model, example_input=inputs["pixel_values"])
@ -556,7 +559,7 @@ As discussed before, the model consists of several blocks which can be
reused for building pipelines for different tasks. In the diagram below,
you can see how image captioning works:
|image01|
|image6|
The visual model accepts the image preprocessed by ``BlipProcessor`` as
input and produces image embeddings, which are directly passed to the
@ -570,12 +573,12 @@ tokenized by ``BlipProcessor`` are provided to the text encoder and then
multimodal question embedding is passed to the text decoder for
performing generation of answers.
|image11|
|image7|
The next step is implementing both pipelines using OpenVINO models.
.. |image01| image:: https://user-images.githubusercontent.com/29454499/221865836-a56da06e-196d-449c-a5dc-4136da6ab5d5.png
.. |image11| image:: https://user-images.githubusercontent.com/29454499/221868167-d0081add-d9f3-4591-80e7-4753c88c1d0a.png
.. |image6| image:: https://user-images.githubusercontent.com/29454499/221865836-a56da06e-196d-449c-a5dc-4136da6ab5d5.png
.. |image7| image:: https://user-images.githubusercontent.com/29454499/221868167-d0081add-d9f3-4591-80e7-4753c88c1d0a.png
.. code:: ipython3
@ -642,6 +645,8 @@ initial token for decoder work.
.. code:: ipython3
if not Path("./blip_model.py").exists():
download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/blip-visual-language-processing/blip_model.py")
from blip_model import OVBlipModel
ov_model = OVBlipModel(model.config, model.decoder_start_token_id, ov_vision_model, ov_text_encoder, text_decoder)

790
docs/notebooks/code-language-id-with-output.rst
View File

@ -1,790 +0,0 @@
Programming Language Classification with OpenVINO
=================================================
Overview
--------
This tutorial will be divided in 2 parts: 1. Create a simple inference
pipeline with a pre-trained model using the OpenVINO™ IR format. 2.
Conduct `post-training
quantization <https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/quantizing-models-post-training.html>`__
on a pre-trained model using Hugging Face Optimum and benchmark
performance.
Feel free to use the notebook outline in Jupyter or your IDE for easy
navigation.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Introduction <#introduction>`__
- `Task <#task>`__
- `Model <#model>`__
- `Part 1: Inference pipeline with
OpenVINO <#part-1-inference-pipeline-with-openvino>`__
- `Install prerequisites <#install-prerequisites>`__
- `Imports <#imports>`__
- `Setting up HuggingFace cache <#setting-up-huggingface-cache>`__
- `Select inference device <#select-inference-device>`__
- `Download resources <#download-resources>`__
- `Create inference pipeline <#create-inference-pipeline>`__
- `Inference on new input <#inference-on-new-input>`__
- `Part 2: OpenVINO post-training quantization with HuggingFace
Optimum <#part-2-openvino-post-training-quantization-with-huggingface-optimum>`__
- `Define constants and
functions <#define-constants-and-functions>`__
- `Load resources <#load-resources>`__
- `Load calibration dataset <#load-calibration-dataset>`__
- `Quantize model <#quantize-model>`__
- `Load quantized model <#load-quantized-model>`__
- `Inference on new input using quantized
model <#inference-on-new-input-using-quantized-model>`__
- `Load evaluation set <#load-evaluation-set>`__
- `Evaluate model <#evaluate-model>`__
- `Additional resources <#additional-resources>`__
- `Clean up <#clean-up>`__
Introduction
------------
Task
~~~~
**Programming language classification** is the task of identifying which
programming language is used in an arbitrary code snippet. This can be
useful to label new data to include in a dataset, and potentially serve
as an intermediary step when input snippets need to be process based on
their programming language.
It is a relatively easy machine learning task given that each
programming language has its own formal symbols, syntax, and grammar.
However, there are some potential edge cases: - **Ambiguous short
snippets**: For example, TypeScript is a superset of JavaScript, meaning
it does everything JavaScript can and more. For a short input snippet,
it might be impossible to distinguish between the two. Given we know
TypeScript is a superset, and the model doesn’t, we should default to
classifying the input as JavaScript in a post-processing step. -
**Nested programming languages**: Some languages are typically used in
tandem. For example, most HTML contains CSS and JavaScript, and it is
not uncommon to see SQL nested in other scripting languages. For such
input, it is unclear what the expected output class should be. -
**Evolving programming language**: Even though programming languages are
formal, their symbols, syntax, and grammar can be revised and updated.
For example, the walrus operator (``:=``) was a symbol distinctively
used in Golang, but was later introduced in Python 3.8.
Model
~~~~~
The classification model that will be used in this notebook is
`CodeBERTa-language-id <https://huggingface.co/huggingface/CodeBERTa-language-id>`__
by HuggingFace. This model was fine-tuned from the masked language
modeling model
`CodeBERTa-small-v1 <https://huggingface.co/huggingface/CodeBERTa-small-v1>`__
trained on the
`CodeSearchNet <https://huggingface.co/huggingface/CodeBERTa-small-v1>`__
dataset (Husain, 2019).
It supports 6 programming languages: - Go - Java - JavaScript - PHP -
Python - Ruby
Part 1: Inference pipeline with OpenVINO
----------------------------------------
For this section, we will use the `HuggingFace
Optimum <https://huggingface.co/docs/optimum/index>`__ library, which
aims to optimize inference on specific hardware and integrates with the
OpenVINO toolkit. The code will be very similar to the `HuggingFace
Transformers <https://huggingface.co/docs/transformers/index>`__, but
will allow to automatically convert models to the OpenVINO™ IR format.
Install prerequisites
~~~~~~~~~~~~~~~~~~~~~
First, complete the `repository installation steps <../../README.md>`__.
Then, the following cell will install: - HuggingFace Optimum with
OpenVINO support - HuggingFace Evaluate to benchmark results
.. code:: ipython3
%pip install -q "diffusers>=0.17.1" "openvino>=2023.1.0" "nncf>=2.5.0" "gradio>=4.19" "onnx>=1.11.0" "transformers>=4.33.0" "torch>=2.1" "evaluate" --extra-index-url https://download.pytorch.org/whl/cpu
%pip install -q "git+https://github.com/huggingface/optimum-intel.git"
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
Imports
~~~~~~~
The import ``OVModelForSequenceClassification`` from Optimum is
equivalent to ``AutoModelForSequenceClassification`` from Transformers
.. code:: ipython3
from functools import partial
from pathlib import Path
import pandas as pd
from datasets import load_dataset, Dataset
import evaluate
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
from optimum.intel import OVModelForSequenceClassification
from optimum.intel.openvino import OVConfig, OVQuantizer, OVWeightQuantizationConfig
from huggingface_hub.utils import RepositoryNotFoundError
.. parsed-literal::
2024-04-17 23:25:25.155476: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:25:25.190609: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:25:25.785655: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
Setting up HuggingFace cache
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resources from HuggingFace will be downloaded in the local folder
``./model`` (next to this notebook) instead of the device global cache
for easy cleanup. Learn more
`here <https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#cache-setup>`__.
.. code:: ipython3
MODEL_NAME = "CodeBERTa-language-id"
MODEL_ID = f"huggingface/{MODEL_NAME}"
MODEL_LOCAL_PATH = Path("./model").joinpath(MODEL_NAME)
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
import ipywidgets as widgets
import openvino as ov
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="AUTO",
description="Device:",
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
Download resources
~~~~~~~~~~~~~~~~~~
.. code:: ipython3
# try to load resources locally
try:
model = OVModelForSequenceClassification.from_pretrained(MODEL_LOCAL_PATH, device=device.value)
tokenizer = AutoTokenizer.from_pretrained(MODEL_LOCAL_PATH)
print(f"Loaded resources from local path: {MODEL_LOCAL_PATH.absolute()}")
# if not found, download from HuggingFace Hub then save locally
except (RepositoryNotFoundError, OSError):
print("Downloading resources from HuggingFace Hub")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.save_pretrained(MODEL_LOCAL_PATH)
# export=True is needed to convert the PyTorch model to OpenVINO
model = OVModelForSequenceClassification.from_pretrained(MODEL_ID, export=True, device=device.value)
model.save_pretrained(MODEL_LOCAL_PATH)
print(f"Ressources cached locally at: {MODEL_LOCAL_PATH.absolute()}")
.. parsed-literal::
Downloading resources from HuggingFace Hub
.. parsed-literal::
Framework not specified. Using pt to export the model.
.. parsed-literal::
Some weights of the model checkpoint at huggingface/CodeBERTa-language-id were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
.. parsed-literal::
Overriding 1 configuration item(s)
.. parsed-literal::
- use_cache -> False
.. parsed-literal::
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
.. parsed-literal::
Compiling the model to AUTO ...
.. parsed-literal::
Ressources cached locally at: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/code-language-id/model/CodeBERTa-language-id
Create inference pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
code_classification_pipe = pipeline("text-classification", model=model, tokenizer=tokenizer)
Inference on new input
~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
# change input snippet to test model
input_snippet = "df['speed'] = df.distance / df.time"
output = code_classification_pipe(input_snippet)
print(f"Input snippet:\n {input_snippet}\n")
print(f"Predicted label: {output[0]['label']}")
print(f"Predicted score: {output[0]['score']:.2}")
.. parsed-literal::
Input snippet:
df['speed'] = df.distance / df.time
Predicted label: python
Predicted score: 0.81
Part 2: OpenVINO post-training quantization with HuggingFace Optimum
--------------------------------------------------------------------
In this section, we will quantize a trained model. At a high-level, this
process consists of using lower precision numbers in the model, which
results in a smaller model size and faster inference at the cost of a
potential marginal performance degradation. `Learn
more <https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/quantizing-models-post-training.html>`__.
The HuggingFace Optimum library supports post-training quantization for
OpenVINO. `Learn
more <https://huggingface.co/docs/optimum/main/en/intel/index>`__.
Define constants and functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
QUANTIZED_MODEL_LOCAL_PATH = MODEL_LOCAL_PATH.with_name(f"{MODEL_NAME}-quantized")
DATASET_NAME = "code_search_net"
LABEL_MAPPING = {"go": 0, "java": 1, "javascript": 2, "php": 3, "python": 4, "ruby": 5}
def preprocess_function(examples: dict, tokenizer):
"""Preprocess inputs by tokenizing the `func_code_string` column"""
return tokenizer(
examples["func_code_string"],
padding="max_length",
max_length=tokenizer.model_max_length,
truncation=True,
)
def map_labels(example: dict) -> dict:
"""Convert string labels to integers"""
label_mapping = {
"go": 0,
"java": 1,
"javascript": 2,
"php": 3,
"python": 4,
"ruby": 5,
}
example["language"] = label_mapping[example["language"]]
return example
def get_dataset_sample(dataset_split: str, num_samples: int) -> Dataset:
"""Create a sample with equal representation of each class without downloading the entire data"""
labels = ["go", "java", "javascript", "php", "python", "ruby"]
example_per_label = num_samples // len(labels)
examples = []
for label in labels:
subset = load_dataset("code_search_net", split=dataset_split, name=label, streaming=True)
subset = subset.map(map_labels)
examples.extend([example for example in subset.shuffle().take(example_per_label)])
return Dataset.from_list(examples)
Load resources
~~~~~~~~~~~~~~
NOTE: the base model is loaded using
``AutoModelForSequenceClassification`` from ``Transformers``
.. code:: ipython3
tokenizer = AutoTokenizer.from_pretrained(MODEL_LOCAL_PATH)
base_model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
quantizer = OVQuantizer.from_pretrained(base_model)
quantization_config = OVWeightQuantizationConfig()
ov_config = OVConfig(quantization_config=quantization_config)
.. parsed-literal::
Some weights of the model checkpoint at huggingface/CodeBERTa-language-id were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Load calibration dataset
~~~~~~~~~~~~~~~~~~~~~~~~
The ``get_dataset_sample()`` function will sample up to ``num_samples``,
with an equal number of examples across the 6 programming languages.
NOTE: Uncomment the method below to download and use the full dataset
(5+ Gb).
.. code:: ipython3
calibration_sample = get_dataset_sample(dataset_split="train", num_samples=120)
calibration_sample = calibration_sample.map(partial(preprocess_function, tokenizer=tokenizer))
# calibration_sample = quantizer.get_calibration_dataset(
# DATASET_NAME,
# preprocess_function=partial(preprocess_function, tokenizer=tokenizer),
# num_samples=120,
# dataset_split="train",
# preprocess_batch=True,
# )
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1461: FutureWarning: The repository for code_search_net contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/code_search_net
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
warnings.warn(
.. parsed-literal::
Map: 0%| | 0/120 [00:00<?, ? examples/s]
Quantize model
~~~~~~~~~~~~~~
Calling ``quantizer.quantize(...)`` will iterate through the calibration
dataset to quantize and save the model
.. code:: ipython3
quantizer.quantize(
ov_config=ov_config,
calibration_dataset=calibration_sample,
save_directory=QUANTIZED_MODEL_LOCAL_PATH,
)
.. parsed-literal::
The support of `torch.nn.Module` will be deprecated in a future release of optimum-intel, please use the corresponding `OVModelForXxx` class to load you model.To convert a PyTorch model to OpenVINO, you can set `export=True` when loading your model as `OVModelForXxx.from_pretrained(..., export=True)`
.. parsed-literal::
Passing the argument `library_name` to `get_supported_tasks_for_model_type` is required, but got library_name=None. Defaulting to `transformers`. An error will be raised in a future version of Optimum if `library_name` is not provided.
.. parsed-literal::
INFO:nncf:Statistics of the bitwidth distribution:
+--------------+---------------------------+-----------------------------------+
| Num bits (N) | % all parameters (layers) | % ratio-defining parameters |
| | | (layers) |
+==============+===========================+===================================+
| 8 | 100% (41 / 41) | 100% (41 / 41) |
+--------------+---------------------------+-----------------------------------+
.. parsed-literal::
Output()
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
.. parsed-literal::
Overriding 1 configuration item(s)
.. parsed-literal::
- use_cache -> False
.. parsed-literal::
WARNING:nncf:You are setting `forward` on an NNCF-processed model object.
NNCF relies on custom-wrapping the `forward` call in order to function properly.
Arbitrary adjustments to the forward function on an NNCFNetwork object have undefined behavior.
If you need to replace the underlying forward function of the original model so that NNCF should be using that instead of the original forward function that NNCF saved during the compressed model creation, you can do this by calling:
model.nncf.set_original_unbound_forward(fn)
if `fn` has an unbound 0-th `self` argument, or
with model.nncf.temporary_bound_original_forward(fn): ...
if `fn` already had 0-th `self` argument bound or never had it in the first place.
.. parsed-literal::
WARNING:nncf:You are setting `forward` on an NNCF-processed model object.
NNCF relies on custom-wrapping the `forward` call in order to function properly.
Arbitrary adjustments to the forward function on an NNCFNetwork object have undefined behavior.
If you need to replace the underlying forward function of the original model so that NNCF should be using that instead of the original forward function that NNCF saved during the compressed model creation, you can do this by calling:
model.nncf.set_original_unbound_forward(fn)
if `fn` has an unbound 0-th `self` argument, or
with model.nncf.temporary_bound_original_forward(fn): ...
if `fn` already had 0-th `self` argument bound or never had it in the first place.
.. parsed-literal::
Configuration saved in model/CodeBERTa-language-id-quantized/openvino_config.json
Load quantized model
~~~~~~~~~~~~~~~~~~~~
NOTE: the argument ``export=True`` is not required since the quantized
model is already in the OpenVINO format.
.. code:: ipython3
quantized_model = OVModelForSequenceClassification.from_pretrained(QUANTIZED_MODEL_LOCAL_PATH, device=device.value)
quantized_code_classification_pipe = pipeline("text-classification", model=quantized_model, tokenizer=tokenizer)
.. parsed-literal::
Compiling the model to AUTO ...
Inference on new input using quantized model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
input_snippet = "df['speed'] = df.distance / df.time"
output = quantized_code_classification_pipe(input_snippet)
print(f"Input snippet:\n {input_snippet}\n")
print(f"Predicted label: {output[0]['label']}")
print(f"Predicted score: {output[0]['score']:.2}")
.. parsed-literal::
Input snippet:
df['speed'] = df.distance / df.time
Predicted label: python
Predicted score: 0.81
Load evaluation set
~~~~~~~~~~~~~~~~~~~
NOTE: Uncomment the method below to download and use the full dataset
(5+ Gb).
.. code:: ipython3
validation_sample = get_dataset_sample(dataset_split="validation", num_samples=120)
# validation_sample = load_dataset(DATASET_NAME, split="validation")
Evaluate model
~~~~~~~~~~~~~~
.. code:: ipython3
# This class is needed due to a current limitation of the Evaluate library with multiclass metrics
# ref: https://discuss.huggingface.co/t/combining-metrics-for-multiclass-predictions-evaluations/21792/16
class ConfiguredMetric:
def __init__(self, metric, *metric_args, **metric_kwargs):
self.metric = metric
self.metric_args = metric_args
self.metric_kwargs = metric_kwargs
def add(self, *args, **kwargs):
return self.metric.add(*args, **kwargs)
def add_batch(self, *args, **kwargs):
return self.metric.add_batch(*args, **kwargs)
def compute(self, *args, **kwargs):
return self.metric.compute(*args, *self.metric_args, **kwargs, **self.metric_kwargs)
@property
def name(self):
return self.metric.name
def _feature_names(self):
return self.metric._feature_names()
First, an ``Evaluator`` object for ``text-classification`` and a set of
``EvaluationModule`` are instantiated. Then, the evaluator
``.compute()`` method is called on both the base
``code_classification_pipe`` and the quantized
``quantized_code_classification_pipeline``. Finally, results are
displayed.
.. code:: ipython3
code_classification_evaluator = evaluate.evaluator("text-classification")
# instantiate an object that can contain multiple `evaluate` metrics
metrics = evaluate.combine(
[
ConfiguredMetric(evaluate.load("f1"), average="macro"),
]
)
base_results = code_classification_evaluator.compute(
model_or_pipeline=code_classification_pipe,
data=validation_sample,
input_column="func_code_string",
label_column="language",
label_mapping=LABEL_MAPPING,
metric=metrics,
)
quantized_results = code_classification_evaluator.compute(
model_or_pipeline=quantized_code_classification_pipe,
data=validation_sample,
input_column="func_code_string",
label_column="language",
label_mapping=LABEL_MAPPING,
metric=metrics,
)
results_df = pd.DataFrame.from_records([base_results, quantized_results], index=["base", "quantized"])
results_df
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>f1</th>
<th>total_time_in_seconds</th>
<th>samples_per_second</th>
<th>latency_in_seconds</th>
</tr>
</thead>
<tbody>
<tr>
<th>base</th>
<td>1.0</td>
<td>2.077464</td>
<td>57.762723</td>
<td>0.017312</td>
</tr>
<tr>
<th>quantized</th>
<td>1.0</td>
<td>1.988597</td>
<td>60.344039</td>
<td>0.016572</td>
</tr>
</tbody>
</table>
</div>
Additional resources
--------------------
- `Grammatical Error Correction
with OpenVINO <grammar-correction-with-output.html>`__ -
`Quantize a Hugging Face Question-Answering Model with
OpenVINO <https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb>`__\ \*\*
Clean up
--------
Uncomment and run cell below to delete all resources cached locally in
./model
.. code:: ipython3
# import os
# import shutil
# try:
# shutil.rmtree(path=QUANTIZED_MODEL_LOCAL_PATH)
# shutil.rmtree(path=MODEL_LOCAL_PATH)
# os.remove(path="./compressed_graph.dot")
# os.remove(path="./original_graph.dot")
# except FileNotFoundError:
# print("Directory was already deleted")

56
docs/notebooks/convert-to-openvino-with-output.rst
View File

@ -35,16 +35,8 @@ Table of contents:
.. parsed-literal::
Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
.. parsed-literal::
Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -189,19 +181,13 @@ NLP model from Hugging Face and export it in ONNX format:
.. parsed-literal::
2024-04-17 23:27:23.489620: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:27:23.524697: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-06 23:46:51.110172: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-06 23:46:51.145296: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:27:24.035165: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
2024-05-06 23:46:51.660347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)
@ -474,10 +460,6 @@ To convert a model to OpenVINO IR, use the following API:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
.. parsed-literal::
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@ -521,10 +503,6 @@ documentation.
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
.. parsed-literal::
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@ -558,10 +536,6 @@ conversion API parameter as ``-1`` or ``?`` when using ``ovc``:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
.. parsed-literal::
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@ -606,10 +580,6 @@ sequence length dimension:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
.. parsed-literal::
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@ -694,12 +664,12 @@ frameworks conversion guides.
.. parsed-literal::
2024-04-17 23:27:44.237192: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2024-04-17 23:27:44.237227: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2024-04-17 23:27:44.237231: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2024-04-17 23:27:44.237454: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2024-04-17 23:27:44.237472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2024-04-17 23:27:44.237475: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
2024-05-06 23:47:11.917183: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
2024-05-06 23:47:11.917219: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
2024-05-06 23:47:11.917224: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
2024-05-06 23:47:11.917431: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
2024-05-06 23:47:11.917454: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
2024-05-06 23:47:11.917459: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
Migration from Legacy conversion API

14
docs/notebooks/convnext-classification-with-output.rst
View File

@ -51,20 +51,8 @@ Prerequisites
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -195,7 +183,7 @@ And print results
Predicted Class: 281
Predicted Label: n02123045 tabby, tabby cat
Predicted Probability: 0.5351971983909607
Predicted Probability: 0.5510364174842834
Convert the model to OpenVINO Intermediate representation format

4
docs/notebooks/cross-lingual-books-alignment-with-output.rst
View File

@ -407,12 +407,12 @@ languages. It has the same architecture as the BERT model but has been
trained on a different task: to produce identical embeddings for
translation pairs.
|image01|
|image1|
This makes LaBSE a great choice for our task and it can be reused for
different language pairs still producing good results.
.. |image01| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/627d3a39-7076-479f-a7b1-392f49a0b83e
.. |image1| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/627d3a39-7076-479f-a7b1-392f49a0b83e
.. code:: ipython3

515
docs/notebooks/ct-scan-live-inference-with-output.rst
View File

@ -1,515 +0,0 @@
Live Inference and Benchmark CT-scan Data with OpenVINO™
========================================================
Kidney Segmentation with PyTorch Lightning and OpenVINO™ - Part 4
-----------------------------------------------------------------
This tutorial is a part of a series on how to train, optimize, quantize
and show live inference on a medical segmentation model. The goal is to
accelerate inference on a kidney segmentation model. The
`UNet <https://arxiv.org/abs/1505.04597>`__ model is trained from
scratch, and the data is from
`Kits19 <https://github.com/neheller/kits19>`__.
This tutorial shows how to benchmark performance of the model and show
live inference with async API and MULTI plugin in OpenVINO.
This notebook needs a quantized OpenVINO IR model and images from the
`KiTS-19 <https://github.com/neheller/kits19>`__ dataset, converted to
2D images. (To learn how the model is quantized, see the `Convert and
Quantize a UNet Model and Show Live
Inference <ct-segmentation-quantize-nncf.ipynb>`__ tutorial.)
This notebook provides a pre-trained model, trained for 20 epochs with
the full KiTS-19 frames dataset, which has an F1 score on the validation
set of 0.9. The training code is available in the `PyTorch MONAI
Training <ct-segmentation-quantize-with-output.html>`__
notebook.
For demonstration purposes, this tutorial will download one converted CT
scan to use for inference.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Imports <#imports>`__
- `Settings <#settings>`__
- `Benchmark Model Performance <#benchmark-model-performance>`__
- `Download and Prepare Data <#download-and-prepare-data>`__
- `Show Live Inference <#show-live-inference>`__
- `Load Model and List of Image
Files <#load-model-and-list-of-image-files>`__
- `Prepare images <#prepare-images>`__
- `Specify device <#specify-device>`__
- `Setting callback function <#setting-callback-function>`__
- `Create asynchronous inference queue and perform
it <#create-asynchronous-inference-queue-and-perform-it>`__
.. code:: ipython3
%pip install -q "openvino>=2023.3.0" "monai>=0.9.1" "nncf>=2.8.0" "opencv-python" "tqdm"
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
Imports
-------
.. code:: ipython3
import os
import zipfile
from pathlib import Path
import numpy as np
from monai.transforms import LoadImage
import openvino as ov
from custom_segmentation import SegmentationModel
# Fetch `notebook_utils` module
import requests
r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py")
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
.. parsed-literal::
2024-04-17 23:29:02.693500: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:29:02.729260: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:29:03.313117: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Settings
--------
To use the pre-trained models, set ``IR_PATH`` to
``"pretrained_model/unet44.xml"`` and ``COMPRESSED_MODEL_PATH`` to
``"pretrained_model/quantized_unet44.xml"``. To use a model that you
trained or optimized yourself, adjust the model paths.
.. code:: ipython3
# The directory that contains the IR model (xml and bin) files.
models_dir = Path("pretrained_model")
ir_model_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/FP16-INT8/"
ir_model_name_xml = "quantized_unet_kits19.xml"
ir_model_name_bin = "quantized_unet_kits19.bin"
download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory=models_dir)
download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory=models_dir)
MODEL_PATH = models_dir / ir_model_name_xml
# Uncomment the next line to use the FP16 model instead of the quantized model.
# MODEL_PATH = "pretrained_model/unet_kits19.xml"
.. parsed-literal::
pretrained_model/quantized_unet_kits19.xml: 0%| | 0.00/280k [00:00<?, ?B/s]
.. parsed-literal::
pretrained_model/quantized_unet_kits19.bin: 0%| | 0.00/1.90M [00:00<?, ?B/s]
Benchmark Model Performance
---------------------------
To measure the inference
performance of the IR model, use `Benchmark
Tool <https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-tool.html>`__
- an inference performance measurement tool in OpenVINO. Benchmark tool
is a command-line application that can be run in the notebook with
``! benchmark_app`` or ``%sx benchmark_app`` commands.
**Note**: The ``benchmark_app`` tool is able to measure the
performance of the OpenVINO Intermediate Representation (OpenVINO IR)
models only. For more accurate performance, run ``benchmark_app`` in
a terminal/command prompt after closing other applications. Run
``benchmark_app -m model.xml -d CPU`` to benchmark async inference on
CPU for one minute. Change ``CPU`` to ``GPU`` to benchmark on GPU.
Run ``benchmark_app --help`` to see an overview of all command-line
options.
.. code:: ipython3
core = ov.Core()
# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.
device_list = ["MULTI:CPU,GPU" if "GPU" in core.available_devices else "AUTO"]
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + device_list,
value=device_list[0],
description="Device:",
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
.. code:: ipython3
# Benchmark model
! benchmark_app -m $MODEL_PATH -d $device.value -t 15 -api sync
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
.. parsed-literal::
[ INFO ] Read model took 13.73 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input.1 (node: input.1) : f32 / [...] / [1,1,512,512]
[ INFO ] Model outputs:
[ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input.1 (node: input.1) : f32 / [N,C,H,W] / [1,1,512,512]
[ INFO ] Model outputs:
[ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 303.13 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: pretrained_unet_kits19
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] MULTI_DEVICE_PRIORITIES: CPU
[ INFO ] CPU:
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] ENABLE_HYPER_THREADING: False
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] NETWORK_NAME: pretrained_unet_kits19
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] PERF_COUNT: NO
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input.1'!. This input will be filled with random values!
[ INFO ] Fill input 'input.1' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 25.71 ms
.. parsed-literal::
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1348 iterations
[ INFO ] Duration: 15011.04 ms
[ INFO ] Latency:
[ INFO ] Median: 10.88 ms
[ INFO ] Average: 10.94 ms
[ INFO ] Min: 10.68 ms
[ INFO ] Max: 13.65 ms
[ INFO ] Throughput: 89.80 FPS
Download and Prepare Data
-------------------------
Download one validation video for live inference.
This tutorial reuses the ``KitsDataset`` class that was also used in the
training and quantization notebook that will be released later.
The data is expected in ``BASEDIR``. The ``BASEDIR`` directory should
contain the ``case_00000`` to ``case_00299`` subdirectories. If the data
for the case specified above does not already exist, it will be
downloaded and extracted in the next cell.
.. code:: ipython3
# Directory that contains the CT scan data. This directory should contain subdirectories
# case_00XXX where XXX is between 000 and 299.
BASEDIR = Path("kits19_frames_1")
# The CT scan case number. For example: 16 for data from the case_00016 directory.
# Currently only 117 is supported.
CASE = 117
case_path = BASEDIR / f"case_{CASE:05d}"
if not case_path.exists():
filename = download_file(f"https://storage.openvinotoolkit.org/data/test_data/openvino_notebooks/kits19/case_{CASE:05d}.zip")
with zipfile.ZipFile(filename, "r") as zip_ref:
zip_ref.extractall(path=BASEDIR)
os.remove(filename) # remove zipfile
print(f"Downloaded and extracted data for case_{CASE:05d}")
else:
print(f"Data for case_{CASE:05d} exists")
.. parsed-literal::
case_00117.zip: 0%| | 0.00/5.48M [00:00<?, ?B/s]
.. parsed-literal::
Downloaded and extracted data for case_00117
Show Live Inference
-------------------
To show live inference on the model in the notebook, use the
asynchronous processing feature of OpenVINO Runtime.
If you use a GPU device, with ``device="GPU"`` or
``device="MULTI:CPU,GPU"`` to do inference on an integrated graphics
card, model loading will be slow the first time you run this code. The
model will be cached, so after the first time model loading will be
faster. For more information on OpenVINO Runtime, including Model
Caching, refer to the `OpenVINO API
tutorial <openvino-api-with-output.html>`__.
We will use
`AsyncInferQueue <https://docs.openvino.ai/2024/openvino-workflow/running-inference/integrate-openvino-with-your-application/python-api-exclusives.html#asyncinferqueue>`__
to perform asynchronous inference. It can be instantiated with compiled
model and a number of jobs - parallel execution threads. If you don’t
pass a number of jobs or pass ``0``, then OpenVINO will pick the optimal
number based on your device and heuristics. After acquiring the
inference queue, there are two jobs to do:
- Preprocess the data and push it to the inference queue. The
preprocessing steps will remain the same.
- Tell the inference queue what to do with the model output after the
inference is finished. It is represented by the ``callback`` python
function that takes an inference result and data that we passed to
the inference queue along with the prepared input data
Everything else will be handled by the ``AsyncInferQueue`` instance.
Load Model and List of Image Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load the segmentation model to OpenVINO Runtime with
``SegmentationModel``, based on the Model API from `Open Model
Zoo <https://github.com/openvinotoolkit/open_model_zoo/>`__. This model
implementation includes pre and post processing for the model. For
``SegmentationModel`` this includes the code to create an overlay of the
segmentation mask on the original image/frame. Uncomment the next cell
to see the implementation.
.. code:: ipython3
core = ov.Core()
segmentation_model = SegmentationModel(ie=core, model_path=Path(MODEL_PATH), sigmoid=True, rotate_and_flip=True)
image_paths = sorted(case_path.glob("imaging_frames/*jpg"))
print(f"{case_path.name}, {len(image_paths)} images")
.. parsed-literal::
case_00117, 69 images
Prepare images
~~~~~~~~~~~~~~
Use the ``reader = LoadImage()`` function to read the images in the same
way as in the
`training <ct-segmentation-quantize-with-output.html>`__
tutorial.
.. code:: ipython3
framebuf = []
next_frame_id = 0
reader = LoadImage(image_only=True, dtype=np.uint8)
while next_frame_id < len(image_paths) - 1:
image_path = image_paths[next_frame_id]
image = reader(str(image_path))
framebuf.append(image)
next_frame_id += 1
Specify device
~~~~~~~~~~~~~~
.. code:: ipython3
device
.. parsed-literal::
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
Setting callback function
~~~~~~~~~~~~~~~~~~~~~~~~~
When ``callback`` is set, any job that ends the inference, calls the
Python function. The ``callback`` function must have two arguments: one
is the request that calls the ``callback``, which provides the
``InferRequest`` API; the other is called ``userdata``, which provides
the possibility of passing runtime values.
The ``callback`` function will show the results of inference.
.. code:: ipython3
import cv2
import copy
from IPython import display
from typing import Dict, Any
# Define a callback function that runs every time the asynchronous pipeline completes inference on a frame
def completion_callback(
infer_request: ov.InferRequest,
user_data: Dict[str, Any],
) -> None:
preprocess_meta = user_data["preprocess_meta"]
raw_outputs = {idx: copy.deepcopy(res.data) for idx, (out, res) in enumerate(zip(infer_request.model_outputs, infer_request.output_tensors))}
frame = segmentation_model.postprocess(raw_outputs, preprocess_meta)
_, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
# Create IPython image
i = display.Image(data=encoded_img)
# Display the image in this notebook
display.clear_output(wait=True)
display.display(i)
Create asynchronous inference queue and perform it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
import time
load_start_time = time.perf_counter()
compiled_model = core.compile_model(segmentation_model.net, device.value)
# Create asynchronous inference queue with optimal number of infer requests
infer_queue = ov.AsyncInferQueue(compiled_model)
infer_queue.set_callback(completion_callback)
load_end_time = time.perf_counter()
results = [None] * len(framebuf)
frame_number = 0
# Perform inference on every frame in the framebuffer
start_time = time.time()
for i, input_frame in enumerate(framebuf):
inputs, preprocessing_meta = segmentation_model.preprocess({segmentation_model.net.input(0): input_frame})
infer_queue.start_async(inputs, {"preprocess_meta": preprocessing_meta})
# Wait until all inference requests in the AsyncInferQueue are completed
infer_queue.wait_all()
stop_time = time.time()
# Calculate total inference time and FPS
total_time = stop_time - start_time
fps = len(framebuf) / total_time
time_per_frame = 1 / fps
print(f"Loaded model to {device} in {load_end_time-load_start_time:.2f} seconds.")
print(f"Total time to infer all frames: {total_time:.3f}s")
print(f"Time per frame: {time_per_frame:.6f}s ({fps:.3f} FPS)")
.. image:: ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png
.. parsed-literal::
Loaded model to Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') in 0.26 seconds.
Total time to infer all frames: 2.480s
Time per frame: 0.036477s (27.415 FPS)

3
docs/notebooks/ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c11bf88f1b96b85ae9d8f26f04113364bdde7d822323cee25f6cb5b01bf4d93a
size 48780

228
docs/notebooks/ct-segmentation-quantize-nncf-with-output.rst
View File

@ -39,7 +39,7 @@ This notebook needs a trained UNet model. We provide a pre-trained
model, trained for 20 epochs with the full
`Kits-19 <https://github.com/neheller/kits19>`__ frames dataset, which
has an F1 score on the validation set of 0.9. The training code is
available in `this notebook <pytorch-monai-training.ipynb>`__.
available in `this notebook <pytorch-monai-training-with-output.html>`__.
NNCF for PyTorch models requires a C++ compiler. On Windows, install
`Microsoft Visual Studio
@ -88,9 +88,9 @@ Table of contents:
.. code:: ipython3
import platform
%pip install -q "openvino>=2023.3.0" "monai>=0.9.1" "torchmetrics>=0.11.0" "nncf>=2.8.0" "opencv-python" torch tqdm --extra-index-url https://download.pytorch.org/whl/cpu
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
@ -100,10 +100,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -122,9 +118,9 @@ Imports
import zipfile
from pathlib import Path
from typing import Union
warnings.filterwarnings("ignore", category=UserWarning)
import cv2
import matplotlib.pyplot as plt
import monai
@ -135,29 +131,31 @@ Imports
from monai.transforms import LoadImage
from nncf.common.logging.logger import set_log_level
from torchmetrics import F1Score as F1
set_log_level(logging.ERROR) # Disables all NNCF info and warning messages
from custom_segmentation import SegmentationModel
from async_pipeline import show_live_inference
# Fetch `notebook_utils` module
import requests
set_log_level(logging.ERROR) # Disables all NNCF info and warning messages
# Fetch `notebook_utils` module
r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py")
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
if not Path("./custom_segmentation.py").exists():
download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/ct-segmentation-quantize/custom_segmentation.py")
from custom_segmentation import SegmentationModel
if not Path("./async_pipeline.py").exists():
download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/ct-segmentation-quantize/async_pipeline.py")
from async_pipeline import show_live_inference
.. parsed-literal::
2024-04-17 23:29:37.722637: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:29:37.758897: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-06 23:48:33.144412: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-06 23:48:33.181396: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:29:38.341773: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-06 23:48:33.764576: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -171,9 +169,7 @@ Settings
By default, this notebook will download one CT scan from the KITS19
dataset that will be used for quantization. To use the full dataset, set
``BASEDIR`` to the path of the dataset, as prepared according to the
`Data Preparation <data-preparation-ct-scan.ipynb>`__ notebook.
dataset that will be used for quantization.
.. code:: ipython3
@ -201,13 +197,13 @@ notebook <pytorch-monai-training.ipynb>`__.
state_dict_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/unet_kits19_state_dict.pth"
state_dict_file = download_file(state_dict_url, directory="pretrained_model")
state_dict = torch.load(state_dict_file, map_location=torch.device("cpu"))
new_state_dict = {}
for k, v in state_dict.items():
new_key = k.replace("_model.", "")
new_state_dict[new_key] = v
new_state_dict.pop("loss_function.pos_weight")
model = monai.networks.nets.BasicUNet(spatial_dims=2, in_channels=1, out_channels=1).eval()
model.load_state_dict(new_state_dict)
@ -252,9 +248,15 @@ Download CT-scan Data
print(f"Data for case_{CASE:05d} exists")
.. parsed-literal::
Data for case_00117 exists
case_00117.zip: 0%| | 0.00/5.48M [00:00<?, ?B/s]
.. parsed-literal::
Downloaded and extracted data for case_00117
Configuration
@ -283,8 +285,8 @@ method to display the images in the expected orientation:
def rotate_and_flip(image):
"""Rotate `image` by 90 degrees and flip horizontally"""
return cv2.flip(cv2.rotate(image, rotateCode=cv2.ROTATE_90_CLOCKWISE), flipCode=1)
class KitsDataset:
def __init__(self, basedir: str):
"""
@ -293,35 +295,35 @@ method to display the images in the expected orientation:
with each subdirectory containing directories imaging_frames, with jpg images, and
segmentation_frames with segmentation masks as png files.
See [data-preparation-ct-scan](./data-preparation-ct-scan.ipynb)
:param basedir: Directory that contains the prepared CT scans
"""
masks = sorted(BASEDIR.glob("case_*/segmentation_frames/*png"))
self.basedir = basedir
self.dataset = masks
print(f"Created dataset with {len(self.dataset)} items. " f"Base directory for data: {basedir}")
def __getitem__(self, index):
"""
Get an item from the dataset at the specified index.
:return: (image, segmentation_mask)
"""
mask_path = self.dataset[index]
image_path = str(mask_path.with_suffix(".jpg")).replace("segmentation_frames", "imaging_frames")
# Load images with MONAI's LoadImage to match data loading in training notebook
mask = LoadImage(image_only=True, dtype=np.uint8)(str(mask_path)).numpy()
img = LoadImage(image_only=True, dtype=np.float32)(str(image_path)).numpy()
if img.shape[:2] != (512, 512):
img = cv2.resize(img.astype(np.uint8), (512, 512)).astype(np.float32)
mask = cv2.resize(mask, (512, 512))
input_image = np.expand_dims(img, axis=0)
return input_image, mask
def __len__(self):
return len(self.dataset)
@ -339,10 +341,10 @@ kidney pixels to verify that the annotations look correct:
image_data, mask = next(item for item in dataset if np.count_nonzero(item[1]) > 5000)
# Remove extra image dimension and rotate and flip the image for visualization
image = rotate_and_flip(image_data.squeeze())
# The data loader returns annotations as (index, mask) and mask in shape (H,W)
mask = rotate_and_flip(mask)
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
ax[0].imshow(image, cmap="gray")
ax[1].imshow(mask, cmap="gray");
@ -420,7 +422,7 @@ this notebook.
.. code:: ipython3
fp32_ir_path = MODEL_DIR / Path("unet_kits19_fp32.xml")
fp32_ir_model = ov.convert_model(model, example_input=torch.ones(1, 1, 512, 512, dtype=torch.float32))
ov.save_model(fp32_ir_model, str(fp32_ir_path))
@ -433,11 +435,7 @@ this notebook.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
@ -469,8 +467,8 @@ steps:
"""
images, _ = data_item
return images
data_loader = torch.utils.data.DataLoader(dataset)
calibration_dataset = nncf.Dataset(data_loader, transform_fn)
quantized_model = nncf.quantize(
@ -536,22 +534,18 @@ Convert quantized model to OpenVINO IR model and save it.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:337: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:337: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:345: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:345: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1102: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1116: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!
Mismatched elements: 248412 / 262144 (94.8%)
Greatest absolute difference: 3.334601879119873 at index (0, 0, 345, 29) (up to 1e-05 allowed)
Greatest relative difference: 21800.240266957342 at index (0, 0, 242, 213) (up to 1e-05 allowed)
Mismatched elements: 249823 / 262144 (95.3%)
Greatest absolute difference: 4.744992733001709 at index (0, 0, 242, 231) (up to 1e-05 allowed)
Greatest relative difference: 26823.613314473136 at index (0, 0, 124, 22) (up to 1e-05 allowed)
_check_trace(
@ -576,7 +570,7 @@ Compare File Size
fp32_ir_model_size = fp32_ir_path.with_suffix(".bin").stat().st_size / 1024
quantized_model_size = int8_ir_path.with_suffix(".bin").stat().st_size / 1024
print(f"FP32 IR model size: {fp32_ir_model_size:.2f} KB")
print(f"INT8 model size: {quantized_model_size:.2f} KB")
@ -584,7 +578,7 @@ Compare File Size
.. parsed-literal::
FP32 IR model size: 3864.14 KB
INT8 model size: 1940.40 KB
INT8 model size: 1953.48 KB
Select Inference Device
@ -597,16 +591,16 @@ Select Inference Device
core = ov.Core()
# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.
device_list = ["MULTI:CPU,GPU" if "GPU" in core.available_devices else "AUTO"]
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + device_list,
value=device_list[0],
description="Device:",
disabled=False,
)
device
@ -627,7 +621,7 @@ Compare Metrics for the original model and the quantized model to be sure that t
int8_compiled_model = core.compile_model(int8_ir_model, device.value)
int8_f1 = compute_f1(int8_compiled_model, dataset)
print(f"FP32 F1: {fp32_f1:.3f}")
print(f"INT8 F1: {int8_f1:.3f}")
@ -674,18 +668,18 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 8.73 ms
[ INFO ] Read model took 8.87 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [?,?,?,?]
@ -699,11 +693,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [?,1,16..,16..]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 151.10 ms
[ INFO ] Compile model took 148.60 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -724,6 +714,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
@ -733,12 +724,13 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ ERROR ] Input x is dynamic. Provide data shapes!
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 486, in main
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 486, in main
data_queue = get_input_data(paths_to_input, app_inputs_info)
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
raise Exception(f"Input {info.name} is dynamic. Provide data shapes!")
Exception: Input x is dynamic. Provide data shapes!
@ -755,22 +747,18 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ]
[ INFO ]
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
.. parsed-literal::
[ INFO ] Read model took 13.30 ms
[ INFO ] Read model took 10.46 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,1,512,512]
@ -784,11 +772,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 233.12 ms
[ INFO ] Compile model took 253.55 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model49
@ -809,6 +793,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model49
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
@ -818,30 +803,23 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
.. parsed-literal::
[ INFO ] First inference took 31.47 ms
.. parsed-literal::
[ INFO ] First inference took 28.04 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 965 iterations
[ INFO ] Duration: 15011.79 ms
[ INFO ] Count: 969 iterations
[ INFO ] Duration: 15000.48 ms
[ INFO ] Latency:
[ INFO ] Median: 15.31 ms
[ INFO ] Average: 15.36 ms
[ INFO ] Min: 15.02 ms
[ INFO ] Max: 17.48 ms
[ INFO ] Throughput: 64.28 FPS
[ INFO ] Median: 15.24 ms
[ INFO ] Average: 15.28 ms
[ INFO ] Min: 14.97 ms
[ INFO ] Max: 17.08 ms
[ INFO ] Throughput: 64.60 FPS
Visually Compare Inference Results
@ -875,11 +853,11 @@ seed is displayed to enable reproducing specific runs of this cell.
# to binary segmentation masks
def sigmoid(x):
return np.exp(-np.logaddexp(0, -x))
num_images = 4
colormap = "gray"
# Load FP32 and INT8 models
core = ov.Core()
fp_model = core.read_model(fp32_ir_path)
@ -888,18 +866,18 @@ seed is displayed to enable reproducing specific runs of this cell.
compiled_model_int8 = core.compile_model(int8_model, device_name=device.value)
output_layer_fp = compiled_model_fp.output(0)
output_layer_int8 = compiled_model_int8.output(0)
# Create subset of dataset
background_slices = (item for item in dataset if np.count_nonzero(item[1]) == 0)
kidney_slices = (item for item in dataset if np.count_nonzero(item[1]) > 50)
data_subset = random.sample(list(background_slices), 2) + random.sample(list(kidney_slices), 2)
# Set seed to current time. To reproduce specific results, copy the printed seed
# and manually set `seed` to that value.
seed = int(time.time())
random.seed(seed)
print(f"Visualizing results with seed {seed}")
fig, ax = plt.subplots(nrows=num_images, ncols=4, figsize=(24, num_images * 4))
for i, (image, mask) in enumerate(data_subset):
display_image = rotate_and_flip(image.squeeze())
@ -908,13 +886,13 @@ seed is displayed to enable reproducing specific runs of this cell.
input_image = np.expand_dims(image, 0)
res_fp = compiled_model_fp([input_image])
res_int8 = compiled_model_int8([input_image])
# Process inference outputs and convert to binary segementation masks
result_mask_fp = sigmoid(res_fp[output_layer_fp]).squeeze().round().astype(np.uint8)
result_mask_int8 = sigmoid(res_int8[output_layer_int8]).squeeze().round().astype(np.uint8)
result_mask_fp = rotate_and_flip(result_mask_fp)
result_mask_int8 = rotate_and_flip(result_mask_int8)
# Display images, annotations, FP32 result and INT8 result
ax[i, 0].imshow(display_image, cmap=colormap)
ax[i, 1].imshow(target_mask, cmap=colormap)
@ -926,7 +904,7 @@ seed is displayed to enable reproducing specific runs of this cell.
.. parsed-literal::
Visualizing results with seed 1713389447
Visualizing results with seed 1715032183
@ -968,7 +946,7 @@ overlay of the segmentation mask on the original image/frame.
.. code:: ipython3
CASE = 117
segmentation_model = SegmentationModel(ie=core, model_path=int8_ir_path, sigmoid=True, rotate_and_flip=True)
case_path = BASEDIR / f"case_{CASE:05d}"
image_paths = sorted(case_path.glob("imaging_frames/*jpg"))
@ -1009,8 +987,8 @@ performs inference, and displays the results on the frames loaded in
.. parsed-literal::
Loaded model to AUTO in 0.22 seconds.
Total time for 68 frames: 2.70 seconds, fps:25.59
Loaded model to AUTO in 0.23 seconds.
Total time for 68 frames: 2.67 seconds, fps:25.86
References

4
docs/notebooks/ct-segmentation-quantize-nncf-with-output_files/ct-segmentation-quantize-nncf-with-output_37_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:08f7f46aea3e4c64f7bcb4dfe777299e9cb77cf5ec36396e340c1be6e8206aef
size 382874
oid sha256:16b3bd0c7d257596831f1d01b363b5f52edfd6db56c147651414f6ac0f77d958
size 382301

2
docs/notebooks/decidiffusion-image-generation-with-output.rst
View File

@ -89,7 +89,7 @@ install required packages
.. code:: ipython3
%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "diffusers" "transformers" "torch>=2.1" "pillow" "openvino>=2023.1.0" "gradio>=4.19" "datasets>=2.14.6" "huggingface-hub>=0.19.4" "nncf>=2.7.0" "peft==0.6.2"
%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "diffusers" "transformers" "torch>=2.1" "pillow" "openvino>=2023.1.0" "gradio>=4.19" "datasets>=2.14.6" "huggingface-hub>=0.19.4" "nncf>=2.7.0" "peft==0.6.2" "opencv-python"
Prepare DeciDiffusion models for OpenVINO format conversion
-----------------------------------------------------------

1061
docs/notebooks/deep-floyd-if-convert-with-output.rst
View File

File diff suppressed because it is too large Load Diff

3
docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_29_3.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f8a26ce14426df514d34e2a3722ad5c94bea8c89dfe95b66d82ac1efa1e1e7e5
size 10887

3
docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_31_3.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7d5a3615f47a2ff2a0f4d537a4e91556c6cedf1fb213056618c072c9ff8f0979
size 129741

3
docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_41_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bab7f183b691aae5fae3d4cd085d8b1a479e128a52778a91da76f992fcebdade
size 1349882

839
docs/notebooks/deep-floyd-if-optimize-with-output.rst
View File

@ -1,839 +0,0 @@
Post-Training Quantization and Weights Compression of DeepFloyd IF model with NNCF
==================================================================================
The goal of this tutorial is to demonstrate how to speed up the model by
applying 8-bit post-training quantization and weights compression from
`NNCF <https://github.com/openvinotoolkit/nncf/>`__ (Neural Network
Compression Framework) and infer optimized model via OpenVINO™ Toolkit.
**NOTE**: you should run
`deep-floyd-if-convert <deep-floyd-if-convert.ipynb>`__ notebook
first to generate OpenVINO IR model that is used for optimization.
The optimization process contains the following steps: 1. Compress
weights of the converted OpenVINO text encoder from
`notebook <deep-floyd-if-convert.ipynb>`__ with NNCF. 2. Quantize the
converted stage_1 and stage_2 U-Nets from
`notebook <deep-floyd-if-convert.ipynb>`__ with NNCF. 2. Check the model
result using the same input data from the
`notebook <deep-floyd-if-convert.ipynb>`__. 3. Compare model size of
converted and optimized models. 4. Compare performance of converted and
optimized models.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Compress weights <#compress-weights>`__
- `Quantize <#quantize>`__
- `Prepare dataset <#prepare-dataset>`__
- `Quantize first stage U-Net <#quantize-first-stage-u-net>`__
- `Quantize second stage U-Net <#quantize-second-stage-u-net>`__
- `Run optimized OpenVINO model <#run-optimized-openvino-model>`__
- `Compare file sizes <#compare-file-sizes>`__
- `Compare performance time of the converted and optimized
models <#compare-performance-time-of-the-converted-and-optimized-models>`__
Prerequisites
-------------
.. code:: ipython3
%pip install -q datasets "nncf>=2.6.0" "torch>=2.1" tqdm
.. code:: ipython3
import nncf
import torch
import openvino as ov
from diffusers import DiffusionPipeline
from diffusers.utils.pil_utils import pt_to_pil
from pathlib import Path
from typing import Any, List
from utils import TextEncoder, UnetFirstStage, UnetSecondStage
checkpoint_variant = "fp16"
model_dtype = torch.float32
RANDOM_SEED = 42
N_DIFFUSION_STEPS = 50
UNET_2_STEPS = 20
core = ov.Core()
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. code:: ipython3
MODEL_DIR = Path("./models")
TEXT_ENCODER_IR_PATH = MODEL_DIR / "encoder_ir.xml"
UNET_I_IR_PATH = MODEL_DIR / "unet_ir_I.xml"
UNET_II_IR_PATH = MODEL_DIR / "unet_ir_II.xml"
if not (TEXT_ENCODER_IR_PATH.exists() and UNET_I_IR_PATH.exists() and UNET_II_IR_PATH.exists()):
raise RuntimeError("This notebook should be run after deep-floyd-if notebook")
.. code:: ipython3
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="AUTO",
description="Device:",
disabled=False,
)
device
.. parsed-literal::
Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')
Compress weights
----------------
Text encoder model consumes ~22 GB of disk space. To avoid running out
of memory, we suggest using 8-bit weights compression instead of
quantization. An optimized model will show less speed up than a
quantized model, but this will significantly reduce the model footprint.
.. code:: ipython3
%%time
text_encoder = core.read_model(TEXT_ENCODER_IR_PATH)
text_encoder_optimized = nncf.compress_weights(text_encoder)
TEXT_ENCODER_INT8_IR_PATH = Path("_optimized.".join(TEXT_ENCODER_IR_PATH.as_posix().split(".")))
ov.save_model(text_encoder_optimized, TEXT_ENCODER_INT8_IR_PATH)
.. parsed-literal::
2023-10-30 08:36:34.384792: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-10-30 08:36:34.423283: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-30 08:36:35.184200: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
CPU times: user 3min 16s, sys: 58 s, total: 4min 14s
Wall time: 4min 12s
Quantize
--------
Prepare dataset
~~~~~~~~~~~~~~~
DeepFloyd IF consists of a U-Net model for first and second stages.
First stage U-Net generates 64x64 px image based on text prompt, second
stage U-Net generates a 256x256 px image based on image from previous
step. We use a portion of train
`LAION2B <https://huggingface.co/datasets/laion/laion2B-en-aesthetic>`__
dataset from Hugging Face as calibration data. LAION2B is the English
subset of the `LAION5B <https://laion.ai/blog/laion-5b/>`__ dataset,
contains over 2 billion objects.
.. code:: ipython3
import numpy as np
from datasets import load_dataset
np.random.seed(RANDOM_SEED)
def get_negative_prompt():
negative_prompts = [
"amateur",
"blurred",
"deformed",
"disfigured",
"disgusting",
"jpeg artifacts",
"low contrast",
"low quality",
"low saturation",
"mangled",
"morbid",
"mutilated",
"mutation",
"out of frame",
"out of frame",
"ugly",
"uncentered",
"underexposed",
"unreal",
]
num_elements = np.random.randint(2, 6)
random_elements = np.random.choice(negative_prompts, num_elements)
return [" ".join(random_elements)]
def prepare_calibration_data(dataloader, stage_1):
"""
This function prepares calibration data from a dataloader for a specified number of initialization steps.
It iterates over the dataloader, fetching batches and storing the relevant data.
"""
data = []
for batch in dataloader:
prompt = batch["TEXT"]
negative_prompt = get_negative_prompt()
prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
data.append((prompt_embeds, negative_embeds))
return data
def prepare_dataset(stage_1, opt_init_steps=300):
"""
Prepares a text dataset for quantization.
"""
dataset = load_dataset("laion/laion2B-en-aesthetic", streaming=True, split="train")
train_dataset = dataset.shuffle(seed=RANDOM_SEED, buffer_size=1000).take(opt_init_steps)
dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=1)
calibration_data = prepare_calibration_data(dataloader, stage_1)
return calibration_data
.. code:: ipython3
%%time
generator = torch.manual_seed(RANDOM_SEED)
opt_init_steps = 300
selection_prob = 0.5
prompts_number = np.ceil(opt_init_steps // (min(N_DIFFUSION_STEPS, UNET_2_STEPS) * selection_prob))
stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant=checkpoint_variant, torch_dtype=model_dtype)
encoded_prompts = prepare_dataset(stage_1, int(prompts_number))
.. parsed-literal::
safety_checker/model.safetensors not found
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[safety_checker/pytorch_model.bin, watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
```
pip install accelerate
```
.
.. parsed-literal::
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]
.. parsed-literal::
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
/home/ea/work/ov_venv/lib/python3.8/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
.. parsed-literal::
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
.. parsed-literal::
Downloading readme: 0%| | 0.00/56.0 [00:00<?, ?B/s]
.. parsed-literal::
Resolving data files: 0%| | 0/128 [00:00<?, ?it/s]
.. parsed-literal::
CPU times: user 18min 16s, sys: 1min 2s, total: 19min 18s
Wall time: 2min 5s
To collect intermediate model inputs for calibration we should customize
``CompiledModel``.
.. code:: ipython3
class CompiledModelDecorator(ov.CompiledModel):
def __init__(self, compiled_model, prob: float, data_cache: List[Any] = []):
super().__init__(compiled_model)
self.data_cache = data_cache
self.prob = np.clip(prob, 0, 1)
def __call__(self, *args, **kwargs):
if np.random.rand() >= self.prob:
self.data_cache.append(*args)
return super().__call__(*args, **kwargs)
.. code:: ipython3
stage_1.unet = UnetFirstStage(UNET_I_IR_PATH, stage_1.unet.config, dtype=model_dtype, device=device.value)
stage_1.set_progress_bar_config(disable=True)
stage_1_data_cache = []
stage_1.unet.unet_openvino = CompiledModelDecorator(stage_1.unet.unet_openvino, prob=selection_prob, data_cache=stage_1_data_cache)
generator = torch.manual_seed(RANDOM_SEED)
stage_2_inputs = [] # to speed up dataset preparation for stage 2 U-Net we can collect several images below
for data in encoded_prompts:
prompt_embeds, negative_embeds = data
image = stage_1(
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_embeds,
generator=generator,
output_type="pt",
num_inference_steps=N_DIFFUSION_STEPS,
).images
stage_2_inputs.append((image, prompt_embeds, negative_embeds))
if len(stage_1_data_cache) >= opt_init_steps:
break
Quantize first stage U-Net
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%time
ov_model = core.read_model(UNET_I_IR_PATH)
stage_1_calibration_dataset = nncf.Dataset(stage_1_data_cache, lambda x: x)
quantized_model = nncf.quantize(
model=ov_model,
calibration_dataset=stage_1_calibration_dataset,
model_type=nncf.ModelType.TRANSFORMER,
advanced_parameters=nncf.AdvancedQuantizationParameters(smooth_quant_alpha=0.25),
)
UNET_I_INT8_PATH = "_optimized.".join(UNET_I_IR_PATH.as_posix().split("."))
ov.save_model(quantized_model, UNET_I_INT8_PATH)
.. parsed-literal::
Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [01:35<00:00, 3.14it/s]
Applying Smooth Quant: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:04<00:00, 17.55it/s]
Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [05:44<00:00, 1.15s/it]
Applying Fast Bias correction: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 268/268 [00:35<00:00, 7.50it/s]
.. parsed-literal::
CPU times: user 1h 8min 46s, sys: 1min 22s, total: 1h 10min 8s
Wall time: 9min 46s
.. code:: ipython3
%%time
from tqdm.notebook import tqdm
start = len(stage_2_inputs)
for i, data in tqdm(enumerate(encoded_prompts[start:])):
prompt_embeds, negative_embeds = data
image = stage_1(
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_embeds,
generator=generator,
output_type="pt",
num_inference_steps=N_DIFFUSION_STEPS,
).images
stage_2_inputs.append((image, prompt_embeds, negative_embeds))
.. parsed-literal::
0it [00:00, ?it/s]
.. parsed-literal::
CPU times: user 1h 17min 46s, sys: 44.9 s, total: 1h 18min 31s
Wall time: 4min 46s
.. code:: ipython3
%%time
generator = torch.manual_seed(RANDOM_SEED)
opt_init_steps = 300
stage_2 = DiffusionPipeline.from_pretrained(
"DeepFloyd/IF-II-M-v1.0",
text_encoder=None,
variant=checkpoint_variant,
torch_dtype=model_dtype,
)
stage_2.set_progress_bar_config(disable=True)
stage_2.unet = UnetSecondStage(UNET_II_IR_PATH, stage_2.unet.config, dtype=model_dtype, device=device.value)
stage_2_data_cache = []
stage_2.unet.unet_openvino = CompiledModelDecorator(stage_2.unet.unet_openvino, prob=selection_prob, data_cache=stage_2_data_cache)
for data in tqdm(stage_2_inputs):
image, prompt_embeds, negative_embeds = data
image = stage_2(
image=image,
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_embeds,
generator=generator,
output_type="pt",
num_inference_steps=UNET_2_STEPS,
).images
if len(stage_2_data_cache) >= opt_init_steps:
break
.. parsed-literal::
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/model.fp16-00001-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, safety_checker/model.fp16.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
```
pip install accelerate
```
.
.. parsed-literal::
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]
.. parsed-literal::
CPU times: user 6h 28min 3s, sys: 2min 11s, total: 6h 30min 15s
Wall time: 24min 32s
Quantize second stage U-Net
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%time
ov_model = core.read_model(UNET_II_IR_PATH)
calibration_dataset = nncf.Dataset(stage_2_data_cache, lambda x: x)
quantized_model = nncf.quantize(
model=ov_model,
calibration_dataset=calibration_dataset,
model_type=nncf.ModelType.TRANSFORMER,
)
UNET_II_INT8_PATH = "_optimized.".join(UNET_II_IR_PATH.as_posix().split("."))
ov.save_model(quantized_model, UNET_II_INT8_PATH)
.. parsed-literal::
Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [12:02<00:00, 2.41s/it]
Applying Smooth Quant: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [00:03<00:00, 15.80it/s]
Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [34:51<00:00, 6.97s/it]
Applying Fast Bias correction: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 245/245 [00:39<00:00, 6.17it/s]
.. parsed-literal::
CPU times: user 7h 57min 5s, sys: 6min 43s, total: 8h 3min 49s
Wall time: 49min 24s
Run optimized OpenVINO model
----------------------------
Let us check predictions with the optimized OpenVINO DeepFloyd IF model
result using the same input data from the `1st
notebook <deep-floyd-if.ipynb>`__.
.. code:: ipython3
prompt = "ultra close color photo portrait of rainbow owl with deer horns in the woods"
negative_prompt = "blurred unreal uncentered occluded"
.. code:: ipython3
%%time
stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant=checkpoint_variant, torch_dtype=model_dtype)
# Initialize the First Stage U-Net wrapper class
stage_1.unet = UnetFirstStage(UNET_I_INT8_PATH, stage_1.unet.config, dtype=model_dtype, device=device.value)
stage_1.text_encoder = TextEncoder(TEXT_ENCODER_INT8_IR_PATH, dtype=model_dtype, device=device.value)
print("The model has been loaded")
# Generate text embeddings
prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
# Fix PRNG seed
generator = torch.manual_seed(RANDOM_SEED)
# Inference
image = stage_1(
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_embeds,
generator=generator,
output_type="pt",
num_inference_steps=N_DIFFUSION_STEPS,
).images
# Show the image
pt_to_pil(image)[0]
.. parsed-literal::
safety_checker/model.safetensors not found
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[safety_checker/pytorch_model.bin, watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
```
pip install accelerate
```
.
.. parsed-literal::
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]
.. parsed-literal::
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
.. parsed-literal::
The model has been loaded
.. parsed-literal::
0%| | 0/50 [00:00<?, ?it/s]
.. parsed-literal::
CPU times: user 3min 39s, sys: 21 s, total: 4min
Wall time: 58.7 s
.. image:: deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.png
.. code:: ipython3
%%time
stage_2 = DiffusionPipeline.from_pretrained(
"DeepFloyd/IF-II-M-v1.0",
text_encoder=None,
variant=checkpoint_variant,
torch_dtype=model_dtype,
)
# Initialize the Second Stage U-Net wrapper class
stage_2.unet = UnetSecondStage(UNET_II_INT8_PATH, stage_2.unet.config, dtype=model_dtype, device=device.value)
print("The model has been loaded")
image = stage_2(
image=image,
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_embeds,
generator=generator,
output_type="pt",
num_inference_steps=UNET_2_STEPS,
).images
# Show the image
pil_image = pt_to_pil(image)[0]
pil_image
.. parsed-literal::
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/model.fp16-00001-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, safety_checker/model.fp16.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
```
pip install accelerate
```
.
.. parsed-literal::
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]
.. parsed-literal::
The model has been loaded
.. parsed-literal::
0%| | 0/20 [00:00<?, ?it/s]
.. parsed-literal::
CPU times: user 6min 20s, sys: 6.78 s, total: 6min 27s
Wall time: 32.1 s
.. image:: deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.png
.. code:: ipython3
import cv2
import numpy as np
from utils import convert_result_to_image, download_omz_model
# 1032: 4x superresolution, 1033: 3x superresolution
model_name = "single-image-super-resolution-1032"
download_omz_model(model_name, MODEL_DIR)
sr_model_xml_path = MODEL_DIR / f"{model_name}.xml"
model = core.read_model(model=sr_model_xml_path)
model.reshape({0: [1, 3, 256, 256], 1: [1, 3, 1024, 1024]})
compiled_sr_model = core.compile_model(model=model, device_name=device.value)
original_image = np.array(pil_image)
bicubic_image = cv2.resize(src=original_image, dsize=(1024, 1024), interpolation=cv2.INTER_CUBIC)
# Reshape the images from (H,W,C) to (N,C,H,W) as expected by the model.
input_image_original = np.expand_dims(original_image.transpose(2, 0, 1), axis=0)
input_image_bicubic = np.expand_dims(bicubic_image.transpose(2, 0, 1), axis=0)
# Model Inference
result = compiled_sr_model([input_image_original, input_image_bicubic])[compiled_sr_model.output(0)]
img = convert_result_to_image(result)
img
.. parsed-literal::
single-image-super-resolution-1032 already downloaded to models
.. image:: deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.png
..
**NOTE**: Accuracy of quantized models can generally be improved by
increasing calibration dataset size. For U-Net models, you can
collect a more diverse dataset by using a smaller ``selection_prob``
value, but this will increase the dataset collection time.
Compare file sizes
^^^^^^^^^^^^^^^^^^
Let’s calculate the compression rate of the optimized IRs file size
relative to the FP16 OpenVINO models file size
.. code:: ipython3
def calculate_compression_rate(ov_model_path):
fp16_ir_model_size = Path(ov_model_path).with_suffix(".bin").stat().st_size / 1024 / 1024
int8_model_path = "_optimized.".join(ov_model_path.as_posix().split("."))
quantized_model_size = Path(int8_model_path).with_suffix(".bin").stat().st_size / 1024 / 1024
print(f'{ov_model_path.as_posix().split(".")[0]}')
print(f" * FP16 IR model size: {fp16_ir_model_size:.2f} MB")
print(f" * INT8 model size: {quantized_model_size:.2f} MB")
print(f" * Model compression rate: {fp16_ir_model_size / quantized_model_size:.3f}")
.. code:: ipython3
for model_path in [TEXT_ENCODER_IR_PATH, UNET_I_IR_PATH, UNET_II_IR_PATH]:
calculate_compression_rate(model_path)
.. parsed-literal::
models/encoder_ir
* FP16 IR model size: 22006.77 MB
* INT8 model size: 4546.70 MB
* Model compression rate: 4.840
models/unet_ir_I
* FP16 IR model size: 1417.56 MB
* INT8 model size: 355.16 MB
* Model compression rate: 3.991
models/unet_ir_II
* FP16 IR model size: 1758.82 MB
* INT8 model size: 440.49 MB
* Model compression rate: 3.993
Compare performance time of the converted and optimized models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To measure the inference performance of OpenVINO FP16 and INT8 models,
use `Benchmark
Tool <https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-tool.html>`__.
**NOTE**: For more accurate performance, run ``benchmark_app`` in a
terminal/command prompt after closing other applications. Run
``benchmark_app --help`` to see an overview of all command-line
options.
.. code:: ipython3
import re
def get_fps(benchmark_output: str):
parsed_output = [line for line in benchmark_output if "Throughput:" in line]
fps = re.findall(r"\d+\.\d+", parsed_output[0])[0]
return fps
Text encoder
.. code:: ipython3
benchmark_output = !benchmark_app -m $TEXT_ENCODER_IR_PATH -d $device.value -api async
original_fps = get_fps(benchmark_output)
print(f"FP16 Text Encoder Throughput: {original_fps} FPS")
benchmark_output = (
!benchmark_app -m $TEXT_ENCODER_INT8_IR_PATH -d $device.value -api async
)
optimized_fps = get_fps(benchmark_output)
print(f"INT8 Text Encoder Throughput: {optimized_fps} FPS")
print(f"Text encoder speed up: {float(optimized_fps) / float(original_fps)}")
.. parsed-literal::
FP16 Text Encoder Throughput: 0.99 FPS
INT8 Text Encoder Throughput: 2.47 FPS
Text encoder speed up: 2.4949494949494953
First stage UNet
.. code:: ipython3
benchmark_output = !benchmark_app -m $UNET_I_IR_PATH -d $device.value -api async
original_fps = get_fps(benchmark_output)
print(f"FP16 1 stage U-Net Throughput: {original_fps} FPS")
benchmark_output = !benchmark_app -m $UNET_I_INT8_PATH -d $device.value -api async
optimized_fps = get_fps(benchmark_output)
print(f"INT8 1 stage U-Net Throughput: {optimized_fps} FPS")
print(f"1 stage U-Net speed up: {float(optimized_fps) / float(original_fps)}")
.. parsed-literal::
FP16 1 stage U-Net Throughput: 4.65 FPS
INT8 1 stage U-Net Throughput: 12.06 FPS
1 stage U-Net speed up: 2.593548387096774
Second stage UNet
.. code:: ipython3
benchmark_output = !benchmark_app -m $UNET_II_IR_PATH -d $device.value -api async
original_fps = get_fps(benchmark_output)
print(f"FP16 2 stage U-Net Throughput: {original_fps} FPS")
benchmark_output = !benchmark_app -m $UNET_II_INT8_PATH -d $device.value -api async
optimized_fps = get_fps(benchmark_output)
print(f"INT8 2 stage U-Net Throughput: {optimized_fps} FPS")
print(f"2 stage U-Net speed up: {float(optimized_fps) / float(original_fps)}")
.. parsed-literal::
FP16 2 stage U-Net Throughput: 0.28 FPS
INT8 2 stage U-Net Throughput: 0.92 FPS
2 stage U-Net speed up: 3.2857142857142856

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:811dbf6cfc44f8eb6babf8771a3a98edb04aab2a92626add147b09205858dbae
size 2577

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8734c254db111a957ea26abd818829ae86fc04d0a007320bc978e4baf2d60913
size 11369

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2292f0150cb59cb15ecc6719ad2f260b3d366d768c1ced1d3b96327a029a776a
size 23341

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d94733cc071e9db841a5b317e4a91919e608b238ca613df63d6bf26064ff9e0f
size 164221

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:17806ca0d95899c5a82007a9719ef58a43a84ac1843beda3c4825a5d031ed57c
size 210327

3
docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:74a8d612863d32d45ed612993997e6c0d773e540b0f7eb619c543156d0231e29
size 1937722

556
docs/notebooks/depth-anything-with-output.rst
View File

@ -75,501 +75,17 @@ Prerequisites
.. parsed-literal::
Cloning into 'Depth-Anything'...
.. parsed-literal::
remote: Enumerating objects: 421, done.
remote: Counting objects: 0% (1/144)
remote: Counting objects: 1% (2/144)
remote: Counting objects: 2% (3/144)
remote: Counting objects: 3% (5/144)
remote: Counting objects: 4% (6/144)
remote: Counting objects: 5% (8/144)
remote: Counting objects: 6% (9/144)
remote: Counting objects: 7% (11/144)
remote: Counting objects: 8% (12/144)
remote: Counting objects: 9% (13/144)
remote: Counting objects: 10% (15/144)
remote: Counting objects: 11% (16/144)
remote: Counting objects: 12% (18/144)
remote: Counting objects: 13% (19/144)
remote: Counting objects: 14% (21/144)
remote: Counting objects: 15% (22/144)
remote: Counting objects: 16% (24/144)
remote: Counting objects: 17% (25/144)
remote: Counting objects: 18% (26/144)
remote: Counting objects: 19% (28/144)
remote: Counting objects: 20% (29/144)
remote: Counting objects: 21% (31/144)
remote: Counting objects: 22% (32/144)
remote: Counting objects: 23% (34/144)
remote: Counting objects: 24% (35/144)
remote: Counting objects: 25% (36/144)
remote: Counting objects: 26% (38/144)
remote: Counting objects: 27% (39/144)
remote: Counting objects: 28% (41/144)
remote: Counting objects: 29% (42/144)
remote: Counting objects: 30% (44/144)
remote: Counting objects: 31% (45/144)
remote: Counting objects: 32% (47/144)
remote: Counting objects: 33% (48/144)
remote: Counting objects: 34% (49/144)
remote: Counting objects: 35% (51/144)
remote: Counting objects: 36% (52/144)
remote: Counting objects: 37% (54/144)
remote: Counting objects: 38% (55/144)
remote: Counting objects: 39% (57/144)
remote: Counting objects: 40% (58/144)
remote: Counting objects: 41% (60/144)
remote: Counting objects: 42% (61/144)
remote: Counting objects: 43% (62/144)
remote: Counting objects: 44% (64/144)
remote: Counting objects: 45% (65/144)
remote: Counting objects: 46% (67/144)
remote: Counting objects: 47% (68/144)
remote: Counting objects: 48% (70/144)
remote: Counting objects: 49% (71/144)
remote: Counting objects: 50% (72/144)
remote: Counting objects: 51% (74/144)
remote: Counting objects: 52% (75/144)
remote: Counting objects: 53% (77/144)
remote: Counting objects: 54% (78/144)
remote: Counting objects: 55% (80/144)
remote: Counting objects: 56% (81/144)
remote: Counting objects: 57% (83/144)
remote: Counting objects: 58% (84/144)
remote: Counting objects: 59% (85/144)
remote: Counting objects: 60% (87/144)
remote: Counting objects: 61% (88/144)
remote: Counting objects: 62% (90/144)
remote: Counting objects: 63% (91/144)
remote: Counting objects: 64% (93/144)
remote: Counting objects: 65% (94/144)
remote: Counting objects: 66% (96/144)
remote: Counting objects: 67% (97/144)
remote: Counting objects: 68% (98/144)
remote: Counting objects: 69% (100/144)
remote: Counting objects: 70% (101/144)
remote: Counting objects: 71% (103/144)
remote: Counting objects: 72% (104/144)
remote: Counting objects: 73% (106/144)
remote: Counting objects: 74% (107/144)
remote: Counting objects: 75% (108/144)
remote: Counting objects: 76% (110/144)
remote: Counting objects: 77% (111/144)
remote: Counting objects: 78% (113/144)
remote: Counting objects: 79% (114/144)
remote: Counting objects: 80% (116/144)
remote: Counting objects: 81% (117/144)
remote: Counting objects: 82% (119/144)
remote: Counting objects: 83% (120/144)
remote: Counting objects: 84% (121/144)
remote: Counting objects: 85% (123/144)
remote: Counting objects: 86% (124/144)
remote: Counting objects: 87% (126/144)
remote: Counting objects: 88% (127/144)
remote: Counting objects: 89% (129/144)
remote: Counting objects: 90% (130/144)
remote: Counting objects: 91% (132/144)
remote: Counting objects: 92% (133/144)
remote: Counting objects: 93% (134/144)
remote: Counting objects: 94% (136/144)
remote: Counting objects: 95% (137/144)
remote: Counting objects: 96% (139/144)
remote: Counting objects: 97% (140/144)
remote: Counting objects: 98% (142/144)
remote: Counting objects: 99% (143/144)
remote: Counting objects: 100% (144/144)
remote: Counting objects: 100% (144/144), done.
remote: Compressing objects: 0% (1/105)
remote: Compressing objects: 1% (2/105)
remote: Compressing objects: 2% (3/105)
remote: Compressing objects: 3% (4/105)
remote: Compressing objects: 4% (5/105)
remote: Compressing objects: 5% (6/105)
remote: Compressing objects: 6% (7/105)
remote: Compressing objects: 7% (8/105)
remote: Compressing objects: 8% (9/105)
remote: Compressing objects: 9% (10/105)
remote: Compressing objects: 10% (11/105)
remote: Compressing objects: 11% (12/105)
remote: Compressing objects: 12% (13/105)
remote: Compressing objects: 13% (14/105)
remote: Compressing objects: 14% (15/105)
remote: Compressing objects: 15% (16/105)
remote: Compressing objects: 16% (17/105)
remote: Compressing objects: 17% (18/105)
remote: Compressing objects: 18% (19/105)
remote: Compressing objects: 19% (20/105)
remote: Compressing objects: 20% (21/105)
remote: Compressing objects: 21% (23/105)
remote: Compressing objects: 22% (24/105)
remote: Compressing objects: 23% (25/105)
remote: Compressing objects: 24% (26/105)
remote: Compressing objects: 25% (27/105)
remote: Compressing objects: 26% (28/105)
remote: Compressing objects: 27% (29/105)
remote: Compressing objects: 28% (30/105)
remote: Compressing objects: 29% (31/105)
remote: Compressing objects: 30% (32/105)
remote: Compressing objects: 31% (33/105)
remote: Compressing objects: 32% (34/105)
remote: Compressing objects: 33% (35/105)
remote: Compressing objects: 34% (36/105)
remote: Compressing objects: 35% (37/105)
remote: Compressing objects: 36% (38/105)
remote: Compressing objects: 37% (39/105)
remote: Compressing objects: 38% (40/105)
remote: Compressing objects: 39% (41/105)
remote: Compressing objects: 40% (42/105)
remote: Compressing objects: 41% (44/105)
remote: Compressing objects: 42% (45/105)
remote: Compressing objects: 43% (46/105)
remote: Compressing objects: 44% (47/105)
remote: Compressing objects: 45% (48/105)
remote: Compressing objects: 46% (49/105)
remote: Compressing objects: 47% (50/105)
remote: Compressing objects: 48% (51/105)
remote: Compressing objects: 49% (52/105)
remote: Compressing objects: 50% (53/105)
remote: Compressing objects: 51% (54/105)
remote: Compressing objects: 52% (55/105)
remote: Compressing objects: 53% (56/105)
remote: Compressing objects: 54% (57/105)
remote: Compressing objects: 55% (58/105)
remote: Compressing objects: 56% (59/105)
remote: Compressing objects: 57% (60/105)
remote: Compressing objects: 58% (61/105)
remote: Compressing objects: 59% (62/105)
remote: Compressing objects: 60% (63/105)
remote: Compressing objects: 61% (65/105)
remote: Compressing objects: 62% (66/105)
remote: Compressing objects: 63% (67/105)
remote: Compressing objects: 64% (68/105)
remote: Compressing objects: 65% (69/105)
remote: Compressing objects: 66% (70/105)
remote: Compressing objects: 67% (71/105)
remote: Compressing objects: 68% (72/105)
remote: Compressing objects: 69% (73/105)
remote: Compressing objects: 70% (74/105)
remote: Compressing objects: 71% (75/105)
remote: Compressing objects: 72% (76/105)
remote: Compressing objects: 73% (77/105)
remote: Compressing objects: 74% (78/105)
remote: Compressing objects: 75% (79/105)
remote: Compressing objects: 76% (80/105)
remote: Compressing objects: 77% (81/105)
remote: Compressing objects: 78% (82/105)
remote: Compressing objects: 79% (83/105)
remote: Compressing objects: 80% (84/105)
remote: Compressing objects: 81% (86/105)
remote: Compressing objects: 82% (87/105)
remote: Compressing objects: 83% (88/105)
remote: Compressing objects: 84% (89/105)
remote: Compressing objects: 85% (90/105)
remote: Compressing objects: 86% (91/105)
remote: Compressing objects: 87% (92/105)
remote: Compressing objects: 88% (93/105)
remote: Compressing objects: 89% (94/105)
remote: Compressing objects: 90% (95/105)
remote: Compressing objects: 91% (96/105)
remote: Compressing objects: 92% (97/105)
remote: Compressing objects: 93% (98/105)
remote: Compressing objects: 94% (99/105)
remote: Compressing objects: 95% (100/105)
remote: Compressing objects: 96% (101/105)
remote: Compressing objects: 97% (102/105)
remote: Compressing objects: 98% (103/105)
remote: Compressing objects: 99% (104/105)
remote: Compressing objects: 100% (105/105)
remote: Compressing objects: 100% (105/105), done.
.. parsed-literal::
Receiving objects: 0% (1/421)
.. parsed-literal::
Receiving objects: 1% (5/421), 10.76 MiB | 21.55 MiB/s
.. parsed-literal::
Receiving objects: 1% (8/421), 19.25 MiB | 19.12 MiB/s
.. parsed-literal::
Receiving objects: 2% (9/421), 19.25 MiB | 19.12 MiB/s
.. parsed-literal::
Receiving objects: 3% (13/421), 31.03 MiB | 20.47 MiB/s
.. parsed-literal::
Receiving objects: 4% (17/421), 31.03 MiB | 20.47 MiB/s
.. parsed-literal::
Receiving objects: 4% (19/421), 43.39 MiB | 21.41 MiB/s
.. parsed-literal::
Receiving objects: 5% (22/421), 56.34 MiB | 22.23 MiB/s
.. parsed-literal::
Receiving objects: 5% (25/421), 56.34 MiB | 22.23 MiB/s
.. parsed-literal::
Receiving objects: 6% (26/421), 56.34 MiB | 22.23 MiB/s
.. parsed-literal::
Receiving objects: 7% (30/421), 69.25 MiB | 22.74 MiB/s
.. parsed-literal::
Receiving objects: 8% (34/421), 69.25 MiB | 22.74 MiB/s
.. parsed-literal::
Receiving objects: 8% (37/421), 82.28 MiB | 22.91 MiB/s
Receiving objects: 9% (38/421), 82.28 MiB | 22.91 MiB/s
.. parsed-literal::
Receiving objects: 10% (43/421), 110.97 MiB | 24.17 MiB/s
.. parsed-literal::
Receiving objects: 11% (47/421), 110.97 MiB | 24.17 MiB/s
.. parsed-literal::
Receiving objects: 11% (49/421), 124.93 MiB | 24.86 MiB/s
.. parsed-literal::
Receiving objects: 12% (51/421), 124.93 MiB | 24.86 MiB/s
.. parsed-literal::
Receiving objects: 13% (55/421), 124.93 MiB | 24.86 MiB/s
.. parsed-literal::
Receiving objects: 14% (59/421), 139.53 MiB | 26.18 MiB/s
.. parsed-literal::
Receiving objects: 14% (60/421), 152.32 MiB | 26.35 MiB/s
.. parsed-literal::
Receiving objects: 15% (64/421), 167.60 MiB | 26.99 MiB/s
.. parsed-literal::
Receiving objects: 15% (66/421), 167.60 MiB | 26.99 MiB/s
.. parsed-literal::
Receiving objects: 16% (68/421), 183.43 MiB | 27.60 MiB/s
.. parsed-literal::
Receiving objects: 17% (72/421), 183.43 MiB | 27.60 MiB/s
.. parsed-literal::
Receiving objects: 18% (76/421), 183.43 MiB | 27.60 MiB/s
.. parsed-literal::
Receiving objects: 19% (80/421), 200.00 MiB | 28.39 MiB/s
.. parsed-literal::
Receiving objects: 20% (85/421), 200.00 MiB | 28.39 MiB/s
.. parsed-literal::
Receiving objects: 20% (88/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 21% (89/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 22% (93/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 23% (97/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 24% (102/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 25% (106/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 26% (110/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 27% (114/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 28% (118/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 29% (123/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 30% (127/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 31% (131/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 32% (135/421), 216.59 MiB | 29.40 MiB/s
Receiving objects: 33% (139/421), 216.59 MiB | 29.40 MiB/s
.. parsed-literal::
Receiving objects: 34% (144/421), 233.69 MiB | 29.70 MiB/s
.. parsed-literal::
Receiving objects: 35% (148/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 36% (152/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 37% (156/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 38% (160/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 39% (165/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 40% (169/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 41% (173/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 42% (177/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 43% (182/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 44% (186/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 45% (190/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 46% (194/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 47% (198/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 48% (203/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 49% (207/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 50% (211/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 51% (215/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 52% (219/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 53% (224/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 54% (228/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 55% (232/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 56% (236/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 57% (240/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 58% (245/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 59% (249/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 60% (253/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 61% (257/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 62% (262/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 63% (266/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 64% (270/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 65% (274/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 66% (278/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 67% (283/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 68% (287/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 69% (291/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 70% (295/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 71% (299/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 72% (304/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 73% (308/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 74% (312/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 75% (316/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 76% (320/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 77% (325/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 78% (329/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 79% (333/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 80% (337/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 81% (342/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 82% (346/421), 233.69 MiB | 29.70 MiB/s
remote: Total 421 (delta 101), reused 40 (delta 39), pack-reused 277
Receiving objects: 83% (350/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 84% (354/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 85% (358/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 86% (363/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 87% (367/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 88% (371/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 89% (375/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 90% (379/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 91% (384/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 92% (388/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 93% (392/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 94% (396/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 95% (400/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 96% (405/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 97% (409/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 98% (413/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 99% (417/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 100% (421/421), 233.69 MiB | 29.70 MiB/s
Receiving objects: 100% (421/421), 237.89 MiB | 27.05 MiB/s, done.
Resolving deltas: 0% (0/144)
Resolving deltas: 1% (2/144)
Resolving deltas: 2% (3/144)
Resolving deltas: 4% (6/144)
Resolving deltas: 5% (8/144)
Resolving deltas: 6% (10/144)
Resolving deltas: 7% (11/144)
Resolving deltas: 8% (12/144)
Resolving deltas: 9% (14/144)
Resolving deltas: 13% (19/144)
Resolving deltas: 16% (24/144)
Resolving deltas: 21% (31/144)
Resolving deltas: 25% (36/144)
Resolving deltas: 27% (39/144)
Resolving deltas: 28% (41/144)
Resolving deltas: 29% (42/144)
Resolving deltas: 40% (58/144)
Resolving deltas: 41% (60/144)
Resolving deltas: 43% (63/144)
Resolving deltas: 46% (67/144)
Resolving deltas: 49% (71/144)
Resolving deltas: 50% (72/144)
Resolving deltas: 65% (94/144)
Resolving deltas: 66% (96/144)
Resolving deltas: 70% (102/144)
Resolving deltas: 71% (103/144)
Resolving deltas: 72% (104/144)
Resolving deltas: 74% (107/144)
Resolving deltas: 77% (111/144)
Resolving deltas: 78% (113/144)
Resolving deltas: 81% (118/144)
Resolving deltas: 83% (120/144)
Resolving deltas: 84% (121/144)
Resolving deltas: 100% (144/144)
Resolving deltas: 100% (144/144), done.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything
.. parsed-literal::
remote: Counting objects: 100% (144/144), done.
remote: Compressing objects: 100% (105/105), done.
remote: Total 421 (delta 101), reused 43 (delta 39), pack-reused 277
Receiving objects: 100% (421/421), 237.89 MiB | 26.48 MiB/s, done.
Resolving deltas: 100% (144/144), done.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything
Note: you may need to restart the kernel to use updated packages.
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -617,10 +133,6 @@ DepthAnything family.
.. parsed-literal::
xFormers not available
.. parsed-literal::
xFormers not available
@ -761,17 +273,13 @@ loading on device using ``core.complie_model``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert H % patch_H == 0, f"Input image height {H} is not a multiple of patch height {patch_H}"
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:74: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:74: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert W % patch_W == 0, f"Input image width {W} is not a multiple of patch width: {patch_W}"
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/vision_transformer.py:183: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/vision_transformer.py:183: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if npatch == N and w == h:
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/depth_anything/dpt.py:133: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/depth_anything/dpt.py:133: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
out = F.interpolate(out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True)
@ -1063,7 +571,7 @@ Run inference on video
.. parsed-literal::
Processed 60 frames in 13.27 seconds. Total FPS (including video processing): 4.52.Inference FPS: 10.65
Processed 60 frames in 13.33 seconds. Total FPS (including video processing): 4.50.Inference FPS: 10.52
Video saved to 'output/Coco Walking in Berkeley_depth_anything.mp4'.
@ -1090,7 +598,7 @@ Run inference on video
.. parsed-literal::
Showing video saved at
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
If you cannot see the video in your browser, please click on the following link to download the video
@ -1227,14 +735,10 @@ quantization code below may take some time.
.. parsed-literal::
2024-04-17 23:32:53.792726: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:32:53.826550: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-06 23:52:17.005825: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-06 23:52:17.039208: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:32:54.388126: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-06 23:52:17.604619: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@ -1282,10 +786,6 @@ quantization code below may take some time.
.. parsed-literal::
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
INFO:nncf:48 ignored nodes were found by name in the NNCFGraph
@ -1310,12 +810,6 @@ quantization code below may take some time.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
.. parsed-literal::
@ -1408,10 +902,10 @@ data.
.. parsed-literal::
Processed 60 frames in 12.69 seconds. Total FPS (including video processing): 4.73.Inference FPS: 12.74
Processed 60 frames in 12.84 seconds. Total FPS (including video processing): 4.67.Inference FPS: 12.79
Video saved to 'output/Coco Walking in Berkeley_depth_anything_int8.mp4'.
Showing video saved at
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
If you cannot see the video in your browser, please click on the following link to download the video
@ -1450,8 +944,8 @@ Compare model file size
.. parsed-literal::
FP16 model size: 47.11 MB
INT8 model size: 24.27 MB
Model compression rate: 1.942
INT8 model size: 24.41 MB
Model compression rate: 1.930
Compare inference time of the FP16 and INT8 models
@ -1491,13 +985,9 @@ Tool <https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-to
.. parsed-literal::
FP16 Throughput: 10.76 FPS
.. parsed-literal::
INT8 Throughput: 14.03 FPS
Speed-up: 1.30
FP16 Throughput: 10.69 FPS
INT8 Throughput: 14.04 FPS
Speed-up: 1.31
Interactive demo

4
docs/notebooks/depth-anything-with-output_files/depth-anything-with-output_44_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb9844273e551b51709586fdf18ebf8558b840a8773950e4fe6d450910e95554
size 451105
oid sha256:5c42b9045954f576a8bdcc0318d85ee8132c8086e27359768e6e5e8158c412de
size 451541

1249
docs/notebooks/depth-estimation-videpth-with-output.rst
View File

File diff suppressed because it is too large Load Diff

3
docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_48_2.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ee26ee6b9040afd4f12332f42ac782d16f05d5126356779401404bf0eed54316
size 215774

3
docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_53_2.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b4c07380e9b4d2ed0fcf4cd3933eefa664d0cf28dbc69dcd5754474c4d4382a9
size 190108

8
docs/notebooks/detectron2-to-openvino-with-output.rst
View File

@ -60,15 +60,7 @@ Install required packages for running model
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.

4
docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:32e589c25e3dbe3506f7daff8bf41d86686035b32d3bb85fb342937707eb7c2e
size 58113
oid sha256:5d10e6ad91e4f0d5c9b289d8f112f86c47143a98ce4603fcc99487dcb882d558
size 58513

4
docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1163cb4251bc3314b3a014f0a883967d75e45aef3d7cd1df2701db45c459ef61
size 509139
oid sha256:1a040b682a9984544752dc46565957f9cbc02d450723acb6a9fcfd9a9963ad84
size 509276

4
docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5cfdeeeb8c2dcb115730a7e3de247f28573c7d2c18d1ed88821842333e72edd7
size 54890
oid sha256:3fe0cf1ddc8c8ebbe68052a133d2d8dcc9478b4837f8b2e40a06696df7a04d52
size 57234

4
docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9e1a0544ca93d604126537c34e33e4be04525b1223b3cd712e812c3047838ae
size 458489
oid sha256:73da85bb37edfdfd377d92a77b9f588565f13dede569ce50d652e486f40c8bcd
size 459409

84
docs/notebooks/distilbert-sequence-classification-with-output.rst
View File

@ -36,47 +36,31 @@ Imports
.. parsed-literal::
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Requirement already satisfied: openvino>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2024.0.0)
Requirement already satisfied: transformers in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.39.3)
Requirement already satisfied: torch>=2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.2.2+cpu)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.2)
Requirement already satisfied: numpy>=1.16.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (1.23.5)
Requirement already satisfied: openvino-telemetry>=2023.2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (2024.1.0)
Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (24.0)
.. parsed-literal::
Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (3.13.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.22.2)
Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2024.4.16)
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2.31.0)
Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.15.2)
Requirement already satisfied: safetensors>=0.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.4.3)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (4.11.0)
Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (1.12)
Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1)
Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1.3)
Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (2024.2.0)
.. parsed-literal::
Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1) (2.1.5)
.. parsed-literal::
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2024.2.2)
Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1) (1.3.0)
.. parsed-literal::
Requirement already satisfied: openvino>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2024.1.0)
Requirement already satisfied: transformers in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.40.2)
Requirement already satisfied: torch>=2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.3.0+cpu)
Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.4)
Requirement already satisfied: numpy<2.0.0,>=1.16.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (1.23.5)
Requirement already satisfied: openvino-telemetry>=2023.2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (2024.1.0)
Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (24.0)
Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (3.14.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.23.0)
Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2024.4.28)
Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2.31.0)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.4.3)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (4.11.0)
Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (1.12)
Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1)
Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1.4)
Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (2024.3.1)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1) (2.1.5)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2024.2.2)
Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1) (1.3.0)
Note: you may need to restart the kernel to use updated packages.
@ -103,6 +87,13 @@ model from Hugging Face.
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path=checkpoint)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Initializing the Tokenizer
--------------------------
@ -123,6 +114,13 @@ understand the context of a sentence. Here, we will use
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=checkpoint)
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Convert Model to OpenVINO Intermediate Representation format
------------------------------------------------------------
@ -159,9 +157,9 @@ optimal execution on end-point target devices.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)

369
docs/notebooks/dolly-2-instruction-following-with-output.rst
View File

@ -85,14 +85,17 @@ Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Convert model using Optimum-CLI
tool <#convert-model-using-optimum-cli-tool>`__
- `Compress model weights <#compress-model-weights>`__
- `Select inference device <#select-inference-device>`__
- `Download and Convert Model <#download-and-convert-model>`__
- `NNCF model weights
compression <#nncf-model-weights-compression>`__
- `Weights Compression using
Optimum-CLI <#weights-compression-using-optimum-cli>`__
- `Select model variant and inference
device <#select-model-variant-and-inference-device>`__
- `Instantiate Model using Optimum
Intel <#instantiate-model-using-optimum-intel>`__
- `Create an instruction-following inference
pipeline <#create-an-instruction-following-inference-pipeline>`__
@ -121,16 +124,234 @@ documentation <https://huggingface.co/docs/optimum/intel/inference>`__.
.. code:: ipython3
%pip install -q "diffusers>=0.16.1" "transformers>=4.33.0" "torch>=2.1" "openvino>=2023.2.0" "nncf>=2.6.0" onnx "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
%pip install -q --upgrade "git+https://github.com/huggingface/optimum-intel.git"
%pip unsinstall -q -y openvino openvino-dev openvino-nightly optimum optimum-intel
%pip install -q "diffusers>=0.16.1" "transformers>=4.33.0" "torch>=2.1" "openvino-nightly" "nncf>=2.10.0" onnx "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
%pip install -q "git+https://github.com/huggingface/optimum-intel.git"
Select inference device
~~~~~~~~~~~~~~~~~~~~~~~
Convert model using Optimum-CLI tool
------------------------------------
`Optimum Intel <https://huggingface.co/docs/optimum/intel/index>`__ is
the interface between the
`Transformers <https://huggingface.co/docs/transformers/index>`__ and
`Diffusers <https://huggingface.co/docs/diffusers/index>`__ libraries
and OpenVINO to accelerate end-to-end pipelines on Intel architectures.
It provides ease-to-use cli interface for exporting models to `OpenVINO
Intermediate Representation
(IR) <https://docs.openvino.ai/2024/documentation/openvino-ir-format.html>`__
format.
The command bellow demonstrates basic command for model export with
``optimum-cli``
.. code:: bash
optimum-cli export openvino --model <model_id_or_path> --task <task> <out_dir>
where ``--model`` argument is model id from HuggingFace Hub or local
directory with model (saved using ``.save_pretrained`` method),
``--task`` is one of `supported
task <https://huggingface.co/docs/optimum/exporters/task_manager>`__
that exported model should solve. For LLMs it will be
``text-generation-with-past``. If model initialization requires to use
remote code, ``--trust-remote-code`` flag additionally should be passed.
Compress model weights
----------------------
The `Weights
Compression <https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html>`__
algorithm is aimed at compressing the weights of the models and can be
used to optimize the model footprint and performance of large models
where the size of weights is relatively larger than the size of
activations, for example, Large Language Models (LLM). Compared to INT8
compression, INT4 compression improves performance even more, but
introduces a minor drop in prediction quality.
Weights Compression using Optimum-CLI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also apply fp16, 8-bit or 4-bit weight compression on the
Linear, Convolutional and Embedding layers when exporting your model
with the CLI by setting ``--weight-format`` to respectively fp16, int8
or int4. This type of optimization allows to reduce the memory footprint
and inference latency. By default the quantization scheme for int8/int4
will be
`asymmetric <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#asymmetric-quantization>`__,
to make it
`symmetric <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#symmetric-quantization>`__
you can add ``--sym``.
For INT4 quantization you can also specify the following arguments : -
The ``--group-size`` parameter will define the group size to use for
quantization, -1 it will results in per-column quantization. - The
``--ratio`` parameter controls the ratio between 4-bit and 8-bit
quantization. If set to 0.9, it means that 90% of the layers will be
quantized to int4 while 10% will be quantized to int8.
Smaller group_size and ratio values usually improve accuracy at the
sacrifice of the model size and inference latency.
**Note**: There may be no speedup for INT4/INT8 compressed models on
dGPU.
.. code:: ipython3
from IPython.display import Markdown, display
import ipywidgets as widgets
prepare_int4_model = widgets.Checkbox(
value=True,
description="Prepare INT4 model",
disabled=False,
)
prepare_int8_model = widgets.Checkbox(
value=False,
description="Prepare INT8 model",
disabled=False,
)
prepare_fp16_model = widgets.Checkbox(
value=False,
description="Prepare FP16 model",
disabled=False,
)
display(prepare_int4_model)
display(prepare_int8_model)
display(prepare_fp16_model)
.. parsed-literal::
Checkbox(value=True, description='Prepare INT4 model')
.. parsed-literal::
Checkbox(value=False, description='Prepare INT8 model')
.. parsed-literal::
Checkbox(value=False, description='Prepare FP16 model')
.. code:: ipython3
from pathlib import Path
model_id = "databricks/dolly-v2-3b"
model_path = Path("dolly-v2-3b")
fp16_model_dir = model_path / "FP16"
int8_model_dir = model_path / "INT8_compressed_weights"
int4_model_dir = model_path / "INT4_compressed_weights"
def convert_to_fp16():
if (fp16_model_dir / "openvino_model.xml").exists():
return
fp16_model_dir.mkdir(parents=True, exist_ok=True)
export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format fp16".format(model_id)
export_command = export_command_base + " " + str(fp16_model_dir)
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
def convert_to_int8():
if (int8_model_dir / "openvino_model.xml").exists():
return
int8_model_dir.mkdir(parents=True, exist_ok=True)
export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int8".format(model_id)
export_command = export_command_base + " " + str(int8_model_dir)
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
def convert_to_int4():
if (int4_model_dir / "openvino_model.xml").exists():
return
int4_model_dir.mkdir(parents=True, exist_ok=True)
export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int4".format(model_id)
export_command = export_command_base + " " + str(int4_model_dir)
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
if prepare_fp16_model.value:
convert_to_fp16()
if prepare_int8_model.value:
convert_to_int8()
if prepare_int4_model.value:
convert_to_int4()
.. code:: ipython3
fp16_weights = fp16_model_dir / "openvino_model.bin"
int8_weights = int8_model_dir / "openvino_model.bin"
int4_weights = int4_model_dir / "openvino_model.bin"
if fp16_weights.exists():
print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
if compressed_weights.exists():
print(f"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB")
if compressed_weights.exists() and fp16_weights.exists():
print(f"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}")
.. parsed-literal::
Size of model with INT4 compressed weights is 2154.54 MB
Select model variant and inference device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
.. code:: ipython3
available_models = []
if int4_model_dir.exists():
available_models.append("INT4")
if int8_model_dir.exists():
available_models.append("INT8")
if fp16_model_dir.exists():
available_models.append("FP16")
model_to_run = widgets.Dropdown(
options=available_models,
value=available_models[0],
description="Model to run:",
disabled=False,
)
model_to_run
.. parsed-literal::
Dropdown(description='Model to run:', options=('INT4',), value='INT4')
.. code:: ipython3
import ipywidgets as widgets
@ -152,12 +373,12 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')
Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')
Download and Convert Model
--------------------------
Instantiate Model using Optimum Intel
-------------------------------------
@ -179,14 +400,18 @@ Below is an example of the Dolly model
model_id = "databricks/dolly-v2-3b"
-model = AutoModelForCausalLM.from_pretrained(model_id)
+model = OVModelForCausalLM.from_pretrained(model_id, from_transformers=True)
+model = OVModelForCausalLM.from_pretrained(model_id, export=True)
Model class initialization starts with calling ``from_pretrained``
method. When downloading and converting Transformers model, the
parameter ``export=True`` should be added. For models where size more We
can save the converted model for the next usage with the
``save_pretrained`` method. Tokenizer class and pipelines API are
compatible with Optimum models.
parameter ``export=True`` should be added (as we already converted model
before, we do not need to provide this parameter). We can save the
converted model for the next usage with the ``save_pretrained`` method.
Tokenizer class and pipelines API are compatible with Optimum models.
You can find more details about OpenVINO LLM inference using HuggingFace
Optimum API in `LLM inference
guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__.
.. code:: ipython3
@ -194,27 +419,21 @@ compatible with Optimum models.
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
model_id = "databricks/dolly-v2-3b"
model_path = Path("dolly-v2-3b")
if model_to_run.value == "INT4":
model_dir = int4_model_dir
elif model_to_run.value == "INT8":
model_dir = int8_model_dir
else:
model_dir = fp16_model_dir
print(f"Loading model from {model_dir}")
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_dir)
current_device = device.value
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
if model_path.exists():
ov_model = OVModelForCausalLM.from_pretrained(model_path, device=current_device, ov_config=ov_config)
else:
ov_model = OVModelForCausalLM.from_pretrained(
model_id,
device=current_device,
export=True,
ov_config=ov_config,
load_in_8bit=False,
)
ov_model.half()
ov_model.save_pretrained(model_path)
ov_model = OVModelForCausalLM.from_pretrained(model_dir, device=current_device, ov_config=ov_config)
.. parsed-literal::
@ -225,87 +444,35 @@ compatible with Optimum models.
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-11-17 13:10:43.359093: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-17 13:10:43.398436: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-01 10:43:29.010748: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-01 10:43:29.012724: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 10:43:29.047558: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 10:43:29.048434: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-17 13:10:44.026743: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Compiling the model to CPU ...
NNCF model weights compression
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NNCF `Weights Compression
algorithm <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/CompressWeights.md>`__
compresses weights of a model to ``INT8``. This is an alternative to
`Quantization
algorithm <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/post_training/Quantization.md>`__
that compresses both weights and activations. Weight compression is
effective in optimizing footprint and performance of large models where
the size of weights is significantly larger than the size of
activations, for example, in Large Language Models (LLMs) such as Dolly
2.0. Additionally, Weight Compression usually leads to almost no
accuracy drop.
.. code:: ipython3
to_compress = widgets.Checkbox(
value=True,
description="INT8 Compression",
disabled=False,
)
print("Click on checkbox for enabling / disabling weights compression")
to_compress
2024-05-01 10:43:29.742257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
.. parsed-literal::
Click on checkbox for enabling / disabling weights compression
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
.. parsed-literal::
Checkbox(value=True, description='INT8 Compression')
.. code:: ipython3
import gc
from optimum.intel import OVQuantizer, OVConfig, OVWeightQuantizationConfig
compressed_model_path = Path(f"{model_path}_compressed")
def calculate_compression_rate(model_path_ov, model_path_ov_compressed):
model_size_original = model_path_ov.with_suffix(".bin").stat().st_size / 2**20
model_size_compressed = model_path_ov_compressed.with_suffix(".bin").stat().st_size / 2**20
print(f"* Original IR model size: {model_size_original:.2f} MB")
print(f"* Compressed IR model size: {model_size_compressed:.2f} MB")
print(f"* Model compression rate: {model_size_original / model_size_compressed:.3f}")
if to_compress.value:
if not compressed_model_path.exists():
quantizer = OVQuantizer.from_pretrained(ov_model)
ov_config = OVConfig(quantization_config=OVWeightQuantizationConfig(bits=8))
quantizer.quantize(save_directory=compressed_model_path, ov_config=ov_config)
del quantizer
gc.collect()
calculate_compression_rate(model_path / "openvino_model.xml", compressed_model_path / "openvino_model.xml")
ov_model = OVModelForCausalLM.from_pretrained(compressed_model_path, device=current_device, ov_config=ov_config)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
Python 3.8.18 (you have 3.8.10)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
.. parsed-literal::
* Original IR model size: 5297.21 MB
* Compressed IR model size: 2657.89 MB
* Model compression rate: 1.993
Loading model from dolly-v2-3b/INT4_compressed_weights
.. parsed-literal::

539
docs/notebooks/efficient-sam-with-output.rst
View File

@ -84,15 +84,7 @@ Prerequisites
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -110,398 +102,13 @@ Prerequisites
.. parsed-literal::
Cloning into 'EfficientSAM'...
.. parsed-literal::
remote: Enumerating objects: 424, done.
remote: Counting objects: 1% (1/85)
remote: Counting objects: 2% (2/85)
remote: Counting objects: 3% (3/85)
remote: Counting objects: 4% (4/85)
remote: Counting objects: 5% (5/85)
remote: Counting objects: 7% (6/85)
remote: Counting objects: 8% (7/85)
remote: Counting objects: 9% (8/85)
remote: Counting objects: 10% (9/85)
remote: Counting objects: 11% (10/85)
remote: Counting objects: 12% (11/85)
remote: Counting objects: 14% (12/85)
remote: Counting objects: 15% (13/85)
remote: Counting objects: 16% (14/85)
remote: Counting objects: 17% (15/85)
remote: Counting objects: 18% (16/85)
remote: Counting objects: 20% (17/85)
remote: Counting objects: 21% (18/85)
remote: Counting objects: 22% (19/85)
remote: Counting objects: 23% (20/85)
remote: Counting objects: 24% (21/85)
remote: Counting objects: 25% (22/85)
remote: Counting objects: 27% (23/85)
remote: Counting objects: 28% (24/85)
remote: Counting objects: 29% (25/85)
remote: Counting objects: 30% (26/85)
remote: Counting objects: 31% (27/85)
remote: Counting objects: 32% (28/85)
remote: Counting objects: 34% (29/85)
remote: Counting objects: 35% (30/85)
remote: Counting objects: 36% (31/85)
remote: Counting objects: 37% (32/85)
remote: Counting objects: 38% (33/85)
remote: Counting objects: 40% (34/85)
remote: Counting objects: 41% (35/85)
remote: Counting objects: 42% (36/85)
remote: Counting objects: 43% (37/85)
remote: Counting objects: 44% (38/85)
remote: Counting objects: 45% (39/85)
remote: Counting objects: 47% (40/85)
remote: Counting objects: 48% (41/85)
remote: Counting objects: 49% (42/85)
remote: Counting objects: 50% (43/85)
remote: Counting objects: 51% (44/85)
remote: Counting objects: 52% (45/85)
remote: Counting objects: 54% (46/85)
remote: Counting objects: 55% (47/85)
remote: Counting objects: 56% (48/85)
remote: Counting objects: 57% (49/85)
remote: Counting objects: 58% (50/85)
remote: Counting objects: 60% (51/85)
remote: Counting objects: 61% (52/85)
remote: Counting objects: 62% (53/85)
remote: Counting objects: 63% (54/85)
remote: Counting objects: 64% (55/85)
remote: Counting objects: 65% (56/85)
remote: Counting objects: 67% (57/85)
remote: Counting objects: 68% (58/85)
remote: Counting objects: 69% (59/85)
remote: Counting objects: 70% (60/85)
remote: Counting objects: 71% (61/85)
remote: Counting objects: 72% (62/85)
remote: Counting objects: 74% (63/85)
remote: Counting objects: 75% (64/85)
remote: Counting objects: 76% (65/85)
remote: Counting objects: 77% (66/85)
remote: Counting objects: 78% (67/85)
remote: Counting objects: 80% (68/85)
remote: Counting objects: 81% (69/85)
remote: Counting objects: 82% (70/85)
remote: Counting objects: 83% (71/85)
remote: Counting objects: 84% (72/85)
remote: Counting objects: 85% (73/85)
remote: Counting objects: 87% (74/85)
remote: Counting objects: 88% (75/85)
remote: Counting objects: 89% (76/85)
remote: Counting objects: 90% (77/85)
remote: Counting objects: 91% (78/85)
remote: Counting objects: 92% (79/85)
remote: Counting objects: 94% (80/85)
remote: Counting objects: 95% (81/85)
remote: Counting objects: 96% (82/85)
remote: Counting objects: 97% (83/85)
remote: Counting objects: 98% (84/85)
remote: Counting objects: 100% (85/85)
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 3% (1/33)
remote: Compressing objects: 6% (2/33)
remote: Compressing objects: 9% (3/33)
remote: Compressing objects: 12% (4/33)
remote: Compressing objects: 15% (5/33)
remote: Compressing objects: 18% (6/33)
remote: Compressing objects: 21% (7/33)
remote: Compressing objects: 24% (8/33)
remote: Compressing objects: 27% (9/33)
remote: Compressing objects: 30% (10/33)
remote: Compressing objects: 33% (11/33)
remote: Compressing objects: 36% (12/33)
remote: Compressing objects: 39% (13/33)
remote: Compressing objects: 42% (14/33)
remote: Compressing objects: 45% (15/33)
remote: Compressing objects: 48% (16/33)
remote: Compressing objects: 51% (17/33)
remote: Compressing objects: 54% (18/33)
remote: Compressing objects: 57% (19/33)
remote: Compressing objects: 60% (20/33)
remote: Compressing objects: 63% (21/33)
remote: Compressing objects: 66% (22/33)
remote: Compressing objects: 69% (23/33)
remote: Compressing objects: 72% (24/33)
remote: Compressing objects: 75% (25/33)
remote: Compressing objects: 78% (26/33)
remote: Compressing objects: 81% (27/33)
remote: Compressing objects: 84% (28/33)
remote: Compressing objects: 87% (29/33)
remote: Compressing objects: 90% (30/33)
remote: Compressing objects: 93% (31/33)
remote: Compressing objects: 96% (32/33)
remote: Compressing objects: 100% (33/33)
remote: Compressing objects: 100% (33/33), done.
Receiving objects: 0% (1/424)
Receiving objects: 1% (5/424)
.. parsed-literal::
Receiving objects: 2% (9/424)
Receiving objects: 3% (13/424)
Receiving objects: 4% (17/424)
Receiving objects: 5% (22/424)
.. parsed-literal::
Receiving objects: 5% (24/424), 7.11 MiB | 7.01 MiB/s
.. parsed-literal::
Receiving objects: 5% (24/424), 15.89 MiB | 7.84 MiB/s
.. parsed-literal::
Receiving objects: 5% (24/424), 25.21 MiB | 8.33 MiB/s
.. parsed-literal::
Receiving objects: 5% (24/424), 35.32 MiB | 8.76 MiB/s
.. parsed-literal::
Receiving objects: 6% (26/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 7% (30/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 8% (34/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 9% (39/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 10% (43/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 11% (47/424), 35.32 MiB | 8.76 MiB/s
Receiving objects: 12% (51/424), 35.32 MiB | 8.76 MiB/s
.. parsed-literal::
Receiving objects: 12% (54/424), 46.28 MiB | 9.54 MiB/s
.. parsed-literal::
Receiving objects: 12% (54/424), 58.26 MiB | 10.40 MiB/s
.. parsed-literal::
Receiving objects: 12% (54/424), 70.74 MiB | 11.14 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 70.74 MiB | 11.14 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 81.30 MiB | 11.36 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 93.08 MiB | 11.64 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 105.71 MiB | 11.89 MiB/s
.. parsed-literal::
Receiving objects: 14% (60/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 15% (64/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 16% (68/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 17% (73/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 18% (77/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 19% (81/424), 105.71 MiB | 11.89 MiB/s
.. parsed-literal::
Receiving objects: 20% (85/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 21% (90/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 22% (94/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 23% (98/424), 105.71 MiB | 11.89 MiB/s
Receiving objects: 24% (102/424), 105.71 MiB | 11.89 MiB/s
.. parsed-literal::
Receiving objects: 25% (106/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 26% (111/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 27% (115/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 28% (119/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 29% (123/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 30% (128/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 31% (132/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 32% (136/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 33% (140/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 34% (145/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 35% (149/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 36% (153/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 37% (157/424), 112.21 MiB | 11.99 MiB/s
Receiving objects: 38% (162/424), 112.21 MiB | 11.99 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 118.79 MiB | 12.04 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 132.56 MiB | 12.57 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 147.18 MiB | 13.34 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 159.14 MiB | 13.27 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 172.36 MiB | 13.30 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 186.46 MiB | 13.47 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 201.36 MiB | 13.62 MiB/s
.. parsed-literal::
Receiving objects: 39% (166/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 40% (170/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 41% (174/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 42% (179/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 43% (183/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 44% (187/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 45% (191/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 46% (196/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 47% (200/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 48% (204/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 49% (208/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 50% (212/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 51% (217/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 52% (221/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 53% (225/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 54% (229/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 55% (234/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 56% (238/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 57% (242/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 58% (246/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 59% (251/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 60% (255/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 61% (259/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 62% (263/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 63% (268/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 64% (272/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 65% (276/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 66% (280/424), 208.90 MiB | 13.65 MiB/s
Receiving objects: 67% (285/424), 208.90 MiB | 13.65 MiB/s
.. parsed-literal::
Receiving objects: 67% (288/424), 216.43 MiB | 13.99 MiB/s
.. parsed-literal::
Receiving objects: 67% (288/424), 232.14 MiB | 14.78 MiB/s
.. parsed-literal::
Receiving objects: 68% (289/424), 240.34 MiB | 15.09 MiB/s
Receiving objects: 69% (293/424), 240.34 MiB | 15.09 MiB/s
Receiving objects: 70% (297/424), 240.34 MiB | 15.09 MiB/s
Receiving objects: 71% (302/424), 240.34 MiB | 15.09 MiB/s
Receiving objects: 72% (306/424), 240.34 MiB | 15.09 MiB/s
Receiving objects: 73% (310/424), 240.34 MiB | 15.09 MiB/s
.. parsed-literal::
Receiving objects: 73% (310/424), 248.86 MiB | 15.43 MiB/s
.. parsed-literal::
Receiving objects: 74% (314/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 75% (318/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 76% (323/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 77% (327/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 78% (331/424), 257.49 MiB | 15.77 MiB/s
.. parsed-literal::
Receiving objects: 79% (335/424), 257.49 MiB | 15.77 MiB/s
.. parsed-literal::
Receiving objects: 80% (340/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 81% (344/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 82% (348/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 83% (352/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 84% (357/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 85% (361/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 86% (365/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 87% (369/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 88% (374/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 89% (378/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 90% (382/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 91% (386/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 92% (391/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 93% (395/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 94% (399/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 95% (403/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 96% (408/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 97% (412/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 98% (416/424), 257.49 MiB | 15.77 MiB/s
remote: Total 424 (delta 76), reused 52 (delta 52), pack-reused 339
Receiving objects: 99% (420/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 100% (424/424), 257.49 MiB | 15.77 MiB/s
Receiving objects: 100% (424/424), 262.14 MiB | 12.58 MiB/s, done.
Resolving deltas: 0% (0/246)
Resolving deltas: 4% (10/246)
Resolving deltas: 6% (17/246)
Resolving deltas: 14% (36/246)
Resolving deltas: 18% (46/246)
Resolving deltas: 22% (56/246)
Resolving deltas: 23% (57/246)
Resolving deltas: 26% (64/246)
Resolving deltas: 27% (68/246)
Resolving deltas: 32% (81/246)
Resolving deltas: 36% (89/246)
Resolving deltas: 37% (92/246)
Resolving deltas: 41% (101/246)
.. parsed-literal::
Resolving deltas: 42% (105/246)
Resolving deltas: 44% (110/246)
Resolving deltas: 46% (114/246)
.. parsed-literal::
Resolving deltas: 48% (120/246)
Resolving deltas: 49% (122/246)
Resolving deltas: 52% (128/246)
Resolving deltas: 54% (134/246)
Resolving deltas: 58% (143/246)
Resolving deltas: 62% (153/246)
Resolving deltas: 63% (155/246)
Resolving deltas: 66% (163/246)
Resolving deltas: 67% (165/246)
Resolving deltas: 69% (172/246)
Resolving deltas: 70% (174/246)
Resolving deltas: 88% (217/246)
Resolving deltas: 96% (237/246)
Resolving deltas: 97% (240/246)
Resolving deltas: 98% (243/246)
.. parsed-literal::
Resolving deltas: 99% (245/246)
.. parsed-literal::
Resolving deltas: 100% (246/246)
Resolving deltas: 100% (246/246), done.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 424 (delta 76), reused 52 (delta 52), pack-reused 339
Receiving objects: 100% (424/424), 262.14 MiB | 24.49 MiB/s, done.
Resolving deltas: 100% (246/246), done.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM
Load PyTorch model
@ -756,27 +363,23 @@ disk using ``openvino.save_model``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:220: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:220: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if (
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:163: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:163: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
size = int(math.sqrt(xy_num))
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert size * size == xy_num
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:166: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:166: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if size != h or size != w:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:251: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:251: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert x.shape[2] == num_patches
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if num_pts > self.decoder_max_num_input_points:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:92: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:92: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
elif num_pts < self.decoder_max_num_input_points:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:126: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:126: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if output_w > 0 and output_h > 0:
@ -1040,14 +643,10 @@ architecture type, we should specify ``transformer`` in ``model_type``.
.. parsed-literal::
2024-04-17 23:43:02.085337: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:43:02.119379: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:01:29.399785: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:01:29.433596: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:43:02.732513: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-07 00:01:30.076398: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@ -1095,10 +694,6 @@ architecture type, we should specify ``transformer`` in ``model_type``.
.. parsed-literal::
INFO:nncf:57 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
INFO:nncf:88 ignored nodes were found by name in the NNCFGraph
@ -1123,12 +718,6 @@ architecture type, we should specify ``transformer`` in ``model_type``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
.. parsed-literal::
@ -1224,8 +813,8 @@ Compare quantized model size
.. parsed-literal::
Size of FP16 model is 21.50 MB
Size of INT8 quantized model is 10.96 MB
Compression rate for INT8 model: 1.962
Size of INT8 quantized model is 11.08 MB
Compression rate for INT8 model: 1.941
Compare inference time of the FP16 and INT8 models
@ -1251,22 +840,18 @@ models, we use ``bencmark_app``.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
.. parsed-literal::
[ INFO ] Read model took 42.86 ms
[ INFO ] Read model took 40.36 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] batched_images (node: batched_images) : f32 / [...] / [?,?,?,?]
@ -1286,11 +871,7 @@ models, we use ``bencmark_app``.
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_2) : f32 / [...] / [?,?,?]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 1414.09 ms
[ INFO ] Compile model took 1355.34 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -1298,10 +879,6 @@ models, we use ``bencmark_app``.
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ] MULTI_DEVICE_PRIORITIES: CPU
.. parsed-literal::
[ INFO ] CPU:
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
@ -1315,6 +892,7 @@ models, we use ``bencmark_app``.
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@ -1324,6 +902,7 @@ models, we use ``bencmark_app``.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'batched_images'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'batched_points'!. This input will be filled with random values!
@ -1333,25 +912,17 @@ models, we use ``bencmark_app``.
[ INFO ] Fill input 'batched_point_labels' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in full mode (inputs filling are included in measurement loop).
.. parsed-literal::
[ INFO ] First inference took 644.24 ms
.. parsed-literal::
[ INFO ] First inference took 645.58 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 49 iterations
[ INFO ] Duration: 15719.86 ms
[ INFO ] Duration: 15813.12 ms
[ INFO ] Latency:
[ INFO ] Median: 1890.94 ms
[ INFO ] Average: 1870.83 ms
[ INFO ] Min: 622.00 ms
[ INFO ] Max: 1963.97 ms
[ INFO ] Throughput: 3.12 FPS
[ INFO ] Median: 1908.46 ms
[ INFO ] Average: 1882.77 ms
[ INFO ] Min: 609.20 ms
[ INFO ] Max: 1971.02 ms
[ INFO ] Throughput: 3.10 FPS
.. code:: ipython3
@ -1366,30 +937,22 @@ models, we use ``bencmark_app``.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
.. parsed-literal::
[ INFO ] Read model took 66.17 ms
[ INFO ] Read model took 53.68 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] batched_images (node: batched_images) : f32 / [...] / [?,?,?,?]
[ INFO ] batched_points (node: batched_points) : i64 / [...] / [?,?,?,?]
.. parsed-literal::
[ INFO ] batched_point_labels (node: batched_point_labels) : i64 / [...] / [?,?,?]
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
@ -1405,11 +968,7 @@ models, we use ``bencmark_app``.
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_2) : f32 / [...] / [?,?,?]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 1891.51 ms
[ INFO ] Compile model took 1854.44 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -1430,6 +989,7 @@ models, we use ``bencmark_app``.
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@ -1439,6 +999,7 @@ models, we use ``bencmark_app``.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'batched_images'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'batched_points'!. This input will be filled with random values!
@ -1448,25 +1009,17 @@ models, we use ``bencmark_app``.
[ INFO ] Fill input 'batched_point_labels' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in full mode (inputs filling are included in measurement loop).
.. parsed-literal::
[ INFO ] First inference took 588.40 ms
.. parsed-literal::
[ INFO ] First inference took 596.95 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 55 iterations
[ INFO ] Duration: 16253.02 ms
[ INFO ] Duration: 16443.47 ms
[ INFO ] Latency:
[ INFO ] Median: 1752.95 ms
[ INFO ] Average: 1732.80 ms
[ INFO ] Min: 518.84 ms
[ INFO ] Max: 1804.26 ms
[ INFO ] Throughput: 3.38 FPS
[ INFO ] Median: 1775.29 ms
[ INFO ] Average: 1756.44 ms
[ INFO ] Min: 630.26 ms
[ INFO ] Max: 1849.40 ms
[ INFO ] Throughput: 3.34 FPS
Interactive segmentation demo

4
docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_16_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:53d92c7dd4acfc54d070d038a1b8a86ea51c1033ba17518ea266a7081901086f
size 1260793
oid sha256:f283354c61f2d5c9aea2fce146bd130ae9133059c0d90b6fadea3cb6207466da
size 1261054

4
docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_24_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dda127dcb26fadf81397d1872fc64e0a6af73cccd791912bfac525cb9040c24f
size 1260102
oid sha256:c36f8c8a366405e85e109c43647bb0cf0f0a92d10d7ddeb70c19e87a6818485b
size 1259311

4
docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_35_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e5ca7e36b0b567092b0960c0ce5d48df585bdf87175f045909d4665325dc7de5
size 1263513
oid sha256:0979318f5d2a5bdb8e7deac91c7ab319d24efcca8be6e5612c4dca27a802ca95
size 1262900

23
docs/notebooks/encodec-audio-compression-with-output.rst
View File

@ -59,11 +59,6 @@ Install required dependencies:
%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "openvino>=2023.3.0" "torch>=2.1" "torchaudio>=2.1" "encodec>=0.1.1" "gradio>=4.19" "librosa>=0.8.1" "matplotlib<=3.7" tqdm
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -134,7 +129,7 @@ bandwidth.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
@ -294,7 +289,7 @@ similar as possible to the original.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
@ -394,13 +389,13 @@ with ``ov.save_model``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:60: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:60: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
ideal_length = (math.ceil(n_frames) - 1) * stride + (kernel_size - padding_total)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
max_pad = max(padding_left, padding_right)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if length <= max_pad:
@ -420,11 +415,11 @@ with ``ov.save_model``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:358: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:358: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
quantized_out = torch.tensor(0.0, device=q_indices.device)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:359: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:359: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
for i, indices in enumerate(q_indices):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:103: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:103: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (padding_left + padding_right) <= x.shape[-1]

378
docs/notebooks/fast-segment-anything-with-output.rst
View File

@ -75,25 +75,8 @@ Install requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -106,29 +89,29 @@ Imports
import ipywidgets as widgets
from pathlib import Path
import openvino as ov
import torch
from PIL import Image, ImageDraw
from ultralytics import FastSAM
# Fetch skip_kernel_extension module
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
)
open("skip_kernel_extension.py", "w").write(r.text)
# Fetch `notebook_utils` module
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
%load_ext skip_kernel_extension
FastSAM in Ultralytics
@ -148,7 +131,7 @@ model and generate a segmentation map.
model_name = "FastSAM-x"
model = FastSAM(model_name)
# Run inference on an image
image_uri = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg"
image_uri = download_file(image_uri)
@ -162,92 +145,7 @@ model and generate a segmentation map.
.. parsed-literal::
0%| | 0.00/138M [00:00<?, ?B/s]
.. parsed-literal::
0%| | 296k/138M [00:00<00:47, 3.02MB/s]
.. parsed-literal::
2%|▏ | 2.45M/138M [00:00<00:10, 14.1MB/s]
.. parsed-literal::
5%|▍ | 6.62M/138M [00:00<00:05, 27.3MB/s]
.. parsed-literal::
13%|█▎ | 17.4M/138M [00:00<00:02, 60.8MB/s]
.. parsed-literal::
21%|██ | 28.6M/138M [00:00<00:01, 80.9MB/s]
.. parsed-literal::
29%|██▉ | 39.8M/138M [00:00<00:01, 93.1MB/s]
.. parsed-literal::
37%|███▋ | 50.9M/138M [00:00<00:00, 101MB/s]
.. parsed-literal::
45%|████▍ | 62.0M/138M [00:00<00:00, 106MB/s]
.. parsed-literal::
53%|█████▎ | 73.2M/138M [00:00<00:00, 109MB/s]
.. parsed-literal::
61%|██████ | 83.6M/138M [00:01<00:00, 109MB/s]
.. parsed-literal::
68%|██████▊ | 94.1M/138M [00:01<00:00, 108MB/s]
.. parsed-literal::
76%|███████▌ | 105M/138M [00:01<00:00, 109MB/s]
.. parsed-literal::
83%|████████▎ | 115M/138M [00:01<00:00, 107MB/s]
.. parsed-literal::
91%|█████████ | 125M/138M [00:01<00:00, 96.6MB/s]
.. parsed-literal::
97%|█████████▋| 135M/138M [00:01<00:00, 97.2MB/s]
.. parsed-literal::
100%|██████████| 138M/138M [00:01<00:00, 90.6MB/s]
100%|██████████| 138M/138M [00:01<00:00, 95.8MB/s]
@ -256,19 +154,11 @@ model and generate a segmentation map.
coco_bike.jpg: 0%| | 0.00/182k [00:00<?, ?B/s]
.. parsed-literal::
image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 768x1024 37 objects, 624.0ms
.. parsed-literal::
Speed: 3.1ms preprocess, 624.0ms inference, 27.7ms postprocess per image at shape (1, 3, 768, 1024)
image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 768x1024 37 objects, 628.0ms
Speed: 3.4ms preprocess, 628.0ms inference, 27.9ms postprocess per image at shape (1, 3, 768, 1024)
The model returns segmentation maps for all the objects on the image.
@ -306,33 +196,17 @@ tracing. The FastSAM model itself is based on YOLOv8 model.
.. parsed-literal::
Ultralytics YOLOv8.1.42 🚀 Python-3.8.10 torch-2.2.2+cpu CPU (Intel Core(TM) i9-10920X 3.50GHz)
.. parsed-literal::
Ultralytics YOLOv8.1.42 🚀 Python-3.8.10 torch-2.3.0+cpu CPU (Intel Core(TM) i9-10920X 3.50GHz)
PyTorch: starting from 'FastSAM-x.pt' with input shape (1, 3, 1024, 1024) BCHW and output shape(s) ((1, 37, 21504), (1, 32, 256, 256)) (138.2 MB)
.. parsed-literal::
OpenVINO: starting export with openvino 2024.0.0-14509-34caeefd078-releases/2024/0...
.. parsed-literal::
OpenVINO: starting export with openvino 2024.1.0-15008-f4afc983258-releases/2024/1...
OpenVINO: export success ✅ 6.1s, saved as 'FastSAM-x_openvino_model/' (276.1 MB)
.. parsed-literal::
Export complete (9.0s)
Results saved to /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/fast-segment-anything
Predict: yolo predict task=segment model=FastSAM-x_openvino_model imgsz=1024
Validate: yolo val task=segment model=FastSAM-x_openvino_model imgsz=1024 data=ultralytics/datasets/sa.yaml
Results saved to /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/fast-segment-anything
Predict: yolo predict task=segment model=FastSAM-x_openvino_model imgsz=1024
Validate: yolo val task=segment model=FastSAM-x_openvino_model imgsz=1024 data=ultralytics/datasets/sa.yaml
Visualize: https://netron.app
@ -368,7 +242,7 @@ from the dropdown list:
description="Device:",
disabled=False,
)
device
@ -406,12 +280,12 @@ object, so we need to redefine the magic ``__call__`` method.
def __init__(self, ov_model, device="CPU", stride=32, ov_config=None) -> None:
ov_config = ov_config or {}
self.model = core.compile_model(ov_model, device, ov_config)
self.stride = stride
self.pt = False
self.fp16 = False
self.names = {0: "object"}
def __call__(self, im, **_):
result = self.model(im)
return torch.from_numpy(result[0]), torch.from_numpy(result[1])
@ -424,7 +298,7 @@ pipeline.
ov_config = {}
if "GPU" in device.value or ("AUTO" in device.value and "GPU" in core.available_devices):
ov_config = {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"}
wrapped_model = OVWrapper(
ov_model_path,
device=device.value,
@ -432,23 +306,15 @@ pipeline.
ov_config=ov_config,
)
model.predictor.model = wrapped_model
ov_results = model(image_uri, device=device.value, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9)
.. parsed-literal::
image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 1024x1024 42 objects, 504.2ms
.. parsed-literal::
Speed: 6.5ms preprocess, 504.2ms inference, 31.9ms postprocess per image at shape (1, 3, 1024, 1024)
image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 1024x1024 42 objects, 510.3ms
Speed: 6.8ms preprocess, 510.3ms inference, 34.2ms postprocess per image at shape (1, 3, 1024, 1024)
One can observe the converted model outputs in the next cell, they is
@ -489,7 +355,7 @@ The optimization process contains the following steps:
description="Quantization",
disabled=False,
)
do_quantize
@ -525,20 +391,20 @@ repo <../yolov8-optimization/>`__.
.. code:: ipython3
%%skip not $do_quantize.value
import pickle
from contextlib import contextmanager
from zipfile import ZipFile
import cv2
from tqdm.autonotebook import tqdm
import nncf
COLLECT_CALIBRATION_DATA = False
calibration_data = []
@contextmanager
def calibration_data_collection():
global COLLECT_CALIBRATION_DATA
@ -547,58 +413,58 @@ repo <../yolov8-optimization/>`__.
yield
finally:
COLLECT_CALIBRATION_DATA = False
class NNCFWrapper:
def __init__(self, ov_model, stride=32) -> None:
self.model = core.read_model(ov_model)
self.compiled_model = core.compile_model(self.model, device_name="CPU")
self.stride = stride
self.pt = False
self.fp16 = False
self.names = {0: "object"}
def __call__(self, im, **_):
if COLLECT_CALIBRATION_DATA:
calibration_data.append(im)
result = self.compiled_model(im)
return torch.from_numpy(result[0]), torch.from_numpy(result[1])
# Fetch data from the web and descibe a dataloader
DATA_URL = "https://ultralytics.com/assets/coco128.zip"
OUT_DIR = Path('.')
download_file(DATA_URL, directory=OUT_DIR, show_progress=True)
if not (OUT_DIR / "coco128/images/train2017").exists():
with ZipFile('coco128.zip', "r") as zip_ref:
zip_ref.extractall(OUT_DIR)
class COCOLoader(torch.utils.data.Dataset):
def __init__(self, images_path):
self.images = list(Path(images_path).iterdir())
def __getitem__(self, index):
if isinstance(index, slice):
return [self.read_image(image_path) for image_path in self.images[index]]
return self.read_image(self.images[index])
def read_image(self, image_path):
image = cv2.imread(str(image_path))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
return image
def __len__(self):
return len(self.images)
def collect_calibration_data_for_decoder(model, calibration_dataset_size: int,
calibration_cache_path: Path):
global calibration_data
if not calibration_cache_path.exists():
coco_dataset = COCOLoader(OUT_DIR / 'coco128/images/train2017')
with calibration_data_collection():
@ -610,10 +476,10 @@ repo <../yolov8-optimization/>`__.
else:
with open(calibration_cache_path, "rb") as f:
calibration_data = pickle.load(f)
return calibration_data
def quantize(model, save_model_path: Path, calibration_cache_path: Path,
calibration_dataset_size: int, preset: nncf.QuantizationPreset):
calibration_data = collect_calibration_data_for_decoder(
@ -634,10 +500,10 @@ repo <../yolov8-optimization/>`__.
)
)
ov.save_model(quantized_ov_decoder, save_model_path)
wrapped_model = NNCFWrapper(ov_model_path, stride=model.predictor.model.stride)
model.predictor.model = wrapped_model
calibration_dataset_size = 128
quantized_model_path = Path(f"{model_name}_quantized") / "FastSAM-x.xml"
calibration_cache_path = Path(f"calibration_data/coco{calibration_dataset_size}.pkl")
@ -667,48 +533,16 @@ repo <../yolov8-optimization/>`__.
.. parsed-literal::
INFO:nncf:3 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
INFO:nncf:8 ignored nodes were found by types in the NNCFGraph
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 275 __module.model.22/aten::sigmoid/Sigmoid
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 325 __module.model.22.dfl.conv/aten::_convolution/Convolution
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 351 __module.model.22/aten::sub/Subtract
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 352 __module.model.22/aten::add/Add
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 365 __module.model.22/aten::add/Add_1
378 __module.model.22/aten::div/Divide
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 366 __module.model.22/aten::sub/Subtract_1
.. parsed-literal::
INFO:nncf:Not adding activation input quantizer for operation: 388 __module.model.22/aten::mul/Multiply
INFO:nncf:Not adding activation input quantizer for operation: 389 __module.model.22/aten::mul/Multiply
@ -732,12 +566,6 @@ repo <../yolov8-optimization/>`__.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
return Tensor(self.data * unwrap_tensor_data(other))
.. parsed-literal::
@ -770,15 +598,15 @@ calibration dataset to measure the performance.
.. code:: ipython3
%%skip not $do_quantize.value
import datetime
coco_dataset = COCOLoader(OUT_DIR / 'coco128/images/train2017')
calibration_dataset_size = 128
wrapped_model = OVWrapper(ov_model_path, device=device.value, stride=model.predictor.model.stride)
model.predictor.model = wrapped_model
start_time = datetime.datetime.now()
for image in tqdm(coco_dataset, desc="Measuring inference time"):
model(image, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9, verbose=False)
@ -802,10 +630,10 @@ calibration dataset to measure the performance.
.. code:: ipython3
%%skip not $do_quantize.value
quantized_wrapped_model = OVWrapper(quantized_model_path, device=device.value, stride=model.predictor.model.stride)
model.predictor.model = quantized_wrapped_model
start_time = datetime.datetime.now()
for image in tqdm(coco_dataset, desc="Measuring inference time"):
model(image, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9, verbose=False)
@ -823,9 +651,9 @@ calibration dataset to measure the performance.
.. parsed-literal::
Segmented in 23 seconds
Resulting in 5.57 fps
That is 2.96 times faster!
Segmented in 22 seconds
Resulting in 5.82 fps
That is 3.09 times faster!
Try out the converted pipeline
@ -845,8 +673,8 @@ bounding boxes on input image.
import cv2
import numpy as np
import matplotlib.pyplot as plt
def fast_process(
annotations,
image,
@ -859,12 +687,12 @@ bounding boxes on input image.
):
original_h = image.height
original_w = image.width
if better_quality:
for i, mask in enumerate(annotations):
mask = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_CLOSE, np.ones((3, 3), np.uint8))
annotations[i] = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_OPEN, np.ones((8, 8), np.uint8))
inner_mask = fast_show_mask(
annotations,
plt.gca(),
@ -874,7 +702,7 @@ bounding boxes on input image.
target_height=original_h,
target_width=original_w,
)
if with_contours:
contour_all = []
temp = np.zeros((original_h, original_w, 1))
@ -892,18 +720,18 @@ bounding boxes on input image.
cv2.drawContours(temp, contour_all, -1, (255, 255, 255), 2 // scale)
color = np.array([0 / 255, 0 / 255, 255 / 255, 0.9])
contour_mask = temp / 255 * color.reshape(1, 1, -1)
image = image.convert("RGBA")
overlay_inner = Image.fromarray((inner_mask * 255).astype(np.uint8), "RGBA")
image.paste(overlay_inner, (0, 0), overlay_inner)
if with_contours:
overlay_contour = Image.fromarray((contour_mask * 255).astype(np.uint8), "RGBA")
image.paste(overlay_contour, (0, 0), overlay_contour)
return image
# CPU post process
def fast_show_mask(
annotation,
@ -921,7 +749,7 @@ bounding boxes on input image.
areas = np.sum(annotation, axis=(1, 2))
sorted_indices = np.argsort(areas)[::1]
annotation = annotation[sorted_indices]
index = (annotation != 0).argmax(axis=0)
if random_color:
color = np.random.random((mask_sum, 1, 1, 3))
@ -930,32 +758,32 @@ bounding boxes on input image.
transparency = np.ones((mask_sum, 1, 1, 1)) * 0.6
visual = np.concatenate([color, transparency], axis=-1)
mask_image = np.expand_dims(annotation, -1) * visual
mask = np.zeros((height, weight, 4))
h_indices, w_indices = np.meshgrid(np.arange(height), np.arange(weight), indexing="ij")
indices = (index[h_indices, w_indices], h_indices, w_indices, slice(None))
mask[h_indices, w_indices, :] = mask_image[indices]
if bbox is not None:
x1, y1, x2, y2 = bbox
ax.add_patch(plt.Rectangle((x1, y1), x2 - x1, y2 - y1, fill=False, edgecolor="b", linewidth=1))
if not retinamask:
mask = cv2.resize(mask, (target_width, target_height), interpolation=cv2.INTER_NEAREST)
return mask
.. code:: ipython3
import gradio as gr
examples = [
[image_uri],
["https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/empty_road_mapillary.jpg"],
["https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/wall.jpg"],
]
object_points = []
background_points = []
bbox_points = []
@ -981,14 +809,14 @@ based on user input.
model.predictor.model = quantized_wrapped_model
else:
model.predictor.model = wrapped_model
input_size = int(input_size)
w, h = image.size
scale = input_size / max(w, h)
new_w = int(w * scale)
new_h = int(h * scale)
image = image.resize((new_w, new_h))
results = model(
image,
retina_masks=use_retina,
@ -996,14 +824,14 @@ based on user input.
conf=conf_threshold,
imgsz=input_size,
)
masks = results[0].masks.data
# Calculate annotations
if not (object_points or bbox_points):
annotations = masks.cpu().numpy()
else:
annotations = []
if object_points:
all_points = object_points + background_points
labels = [1] * len(object_points) + [0] * len(background_points)
@ -1026,20 +854,20 @@ based on user input.
x = max(min(x, new_w), 0)
y = max(min(y, new_h), 0)
scaled_bbox_points.append((x, y))
for i in range(0, len(scaled_bbox_points) - 1, 2):
x0, y0, x1, y1 = *scaled_bbox_points[i], *scaled_bbox_points[i + 1]
intersection_area = torch.sum(masks[:, y0:y1, x0:x1], dim=(1, 2))
masks_area = torch.sum(masks, dim=(1, 2))
bbox_area = (y1 - y0) * (x1 - x0)
union = bbox_area + masks_area - intersection_area
iou = intersection_area / union
max_iou_index = torch.argmax(iou)
annotations.append(masks[max_iou_index].cpu().numpy())
return fast_process(
annotations=np.array(annotations),
image=image,
@ -1085,8 +913,8 @@ based on user input.
fill=color,
)
return img
def clear_points() -> (Image.Image, None):
"""Gradio clear points callback."""
global object_points, background_points, bbox_points
@ -1095,8 +923,8 @@ based on user input.
background_points = []
bbox_points = []
return last_image, None
def save_last_picked_image(img: Image.Image) -> None:
"""Gradio callback saves the last used image."""
global last_image
@ -1106,8 +934,8 @@ based on user input.
clear_points()
# Removes the segmentation map output
return None
with gr.Blocks(title="Fast SAM") as demo:
with gr.Row(variant="panel"):
original_img = gr.Image(label="Input", value=examples[0][0], type="pil")
@ -1133,18 +961,18 @@ based on user input.
run_on_click=True,
outputs=segmented_img,
)
# Callbacks
original_img.select(select_point, inputs=[original_img, point_type], outputs=original_img)
original_img.upload(save_last_picked_image, inputs=original_img, outputs=segmented_img)
clear_button.click(clear_points, outputs=[original_img, segmented_img])
segment_button.click(segment, inputs=[original_img, model_type], outputs=segmented_img)
try:
demo.queue().launch(debug=False)
except Exception:
demo.queue().launch(share=True, debug=False)
# If you are launching remotely, specify server_name and server_port
# EXAMPLE: `demo.launch(server_name="your server name", server_port="server port in int")`
# To learn more please refer to the Gradio docs: https://gradio.app/docs/
@ -1153,7 +981,7 @@ based on user input.
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.

1177
docs/notebooks/fastcomposer-image-generation-with-output.rst
View File

File diff suppressed because it is too large Load Diff

3
docs/notebooks/fastcomposer-image-generation-with-output_files/fastcomposer-image-generation-with-output_32_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b2697e38ff8c341e543f2a95696466760526e4e336e5bb07bea44144ea25e693
size 68520

3
docs/notebooks/fastcomposer-image-generation-with-output_files/fastcomposer-image-generation-with-output_32_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4d8a950a39b16715cc486c3f5a6e9f1f393041040e206e2ae3cdc35601618195
size 570952

756
docs/notebooks/grounded-segment-anything-with-output.rst
View File

@ -55,15 +55,6 @@ Clone repositories and install requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -121,693 +112,19 @@ segmentation you can select vanilla ``SAM``.
.. parsed-literal::
Cloning into 'GroundingDINO'...
.. parsed-literal::
remote: Enumerating objects: 379, done.
remote: Counting objects: 0% (1/177)
remote: Counting objects: 1% (2/177)
remote: Counting objects: 2% (4/177)
remote: Counting objects: 3% (6/177)
remote: Counting objects: 4% (8/177)
remote: Counting objects: 5% (9/177)
remote: Counting objects: 6% (11/177)
remote: Counting objects: 7% (13/177)
remote: Counting objects: 8% (15/177)
remote: Counting objects: 9% (16/177)
remote: Counting objects: 10% (18/177)
remote: Counting objects: 11% (20/177)
remote: Counting objects: 12% (22/177)
remote: Counting objects: 13% (24/177)
remote: Counting objects: 14% (25/177)
remote: Counting objects: 15% (27/177)
remote: Counting objects: 16% (29/177)
remote: Counting objects: 17% (31/177)
remote: Counting objects: 18% (32/177)
remote: Counting objects: 19% (34/177)
remote: Counting objects: 20% (36/177)
remote: Counting objects: 21% (38/177)
remote: Counting objects: 22% (39/177)
remote: Counting objects: 23% (41/177)
remote: Counting objects: 24% (43/177)
remote: Counting objects: 25% (45/177)
remote: Counting objects: 26% (47/177)
remote: Counting objects: 27% (48/177)
remote: Counting objects: 28% (50/177)
remote: Counting objects: 29% (52/177)
remote: Counting objects: 30% (54/177)
remote: Counting objects: 31% (55/177)
remote: Counting objects: 32% (57/177)
remote: Counting objects: 33% (59/177)
remote: Counting objects: 34% (61/177)
remote: Counting objects: 35% (62/177)
remote: Counting objects: 36% (64/177)
remote: Counting objects: 37% (66/177)
remote: Counting objects: 38% (68/177)
remote: Counting objects: 39% (70/177)
remote: Counting objects: 40% (71/177)
remote: Counting objects: 41% (73/177)
remote: Counting objects: 42% (75/177)
remote: Counting objects: 43% (77/177)
remote: Counting objects: 44% (78/177)
remote: Counting objects: 45% (80/177)
remote: Counting objects: 46% (82/177)
remote: Counting objects: 47% (84/177)
remote: Counting objects: 48% (85/177)
remote: Counting objects: 49% (87/177)
remote: Counting objects: 50% (89/177)
remote: Counting objects: 51% (91/177)
remote: Counting objects: 52% (93/177)
remote: Counting objects: 53% (94/177)
remote: Counting objects: 54% (96/177)
remote: Counting objects: 55% (98/177)
remote: Counting objects: 56% (100/177)
remote: Counting objects: 57% (101/177)
remote: Counting objects: 58% (103/177)
remote: Counting objects: 59% (105/177)
remote: Counting objects: 60% (107/177)
remote: Counting objects: 61% (108/177)
remote: Counting objects: 62% (110/177)
remote: Counting objects: 63% (112/177)
remote: Counting objects: 64% (114/177)
remote: Counting objects: 65% (116/177)
remote: Counting objects: 66% (117/177)
remote: Counting objects: 67% (119/177)
remote: Counting objects: 68% (121/177)
remote: Counting objects: 69% (123/177)
remote: Counting objects: 70% (124/177)
remote: Counting objects: 71% (126/177)
remote: Counting objects: 72% (128/177)
remote: Counting objects: 73% (130/177)
remote: Counting objects: 74% (131/177)
remote: Counting objects: 75% (133/177)
remote: Counting objects: 76% (135/177)
remote: Counting objects: 77% (137/177)
remote: Counting objects: 78% (139/177)
remote: Counting objects: 79% (140/177)
remote: Counting objects: 80% (142/177)
remote: Counting objects: 81% (144/177)
remote: Counting objects: 82% (146/177)
remote: Counting objects: 83% (147/177)
remote: Counting objects: 84% (149/177)
remote: Counting objects: 85% (151/177)
remote: Counting objects: 86% (153/177)
remote: Counting objects: 87% (154/177)
remote: Counting objects: 88% (156/177)
remote: Counting objects: 89% (158/177)
remote: Counting objects: 90% (160/177)
remote: Counting objects: 91% (162/177)
remote: Counting objects: 92% (163/177)
remote: Counting objects: 93% (165/177)
remote: Counting objects: 94% (167/177)
remote: Counting objects: 95% (169/177)
remote: Counting objects: 96% (170/177)
remote: Counting objects: 97% (172/177)
remote: Counting objects: 98% (174/177)
remote: Counting objects: 99% (176/177)
remote: Counting objects: 100% (177/177)
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 1% (1/64)
remote: Compressing objects: 3% (2/64)
remote: Compressing objects: 4% (3/64)
remote: Compressing objects: 6% (4/64)
remote: Compressing objects: 7% (5/64)
remote: Compressing objects: 9% (6/64)
remote: Compressing objects: 10% (7/64)
remote: Compressing objects: 12% (8/64)
remote: Compressing objects: 14% (9/64)
remote: Compressing objects: 15% (10/64)
remote: Compressing objects: 17% (11/64)
remote: Compressing objects: 18% (12/64)
remote: Compressing objects: 20% (13/64)
remote: Compressing objects: 21% (14/64)
remote: Compressing objects: 23% (15/64)
remote: Compressing objects: 25% (16/64)
remote: Compressing objects: 26% (17/64)
remote: Compressing objects: 28% (18/64)
remote: Compressing objects: 29% (19/64)
remote: Compressing objects: 31% (20/64)
remote: Compressing objects: 32% (21/64)
remote: Compressing objects: 34% (22/64)
remote: Compressing objects: 35% (23/64)
remote: Compressing objects: 37% (24/64)
remote: Compressing objects: 39% (25/64)
remote: Compressing objects: 40% (26/64)
remote: Compressing objects: 42% (27/64)
remote: Compressing objects: 43% (28/64)
remote: Compressing objects: 45% (29/64)
remote: Compressing objects: 46% (30/64)
remote: Compressing objects: 48% (31/64)
remote: Compressing objects: 50% (32/64)
remote: Compressing objects: 51% (33/64)
remote: Compressing objects: 53% (34/64)
remote: Compressing objects: 54% (35/64)
remote: Compressing objects: 56% (36/64)
remote: Compressing objects: 57% (37/64)
remote: Compressing objects: 59% (38/64)
remote: Compressing objects: 60% (39/64)
remote: Compressing objects: 62% (40/64)
remote: Compressing objects: 64% (41/64)
remote: Compressing objects: 65% (42/64)
remote: Compressing objects: 67% (43/64)
remote: Compressing objects: 68% (44/64)
remote: Compressing objects: 70% (45/64)
remote: Compressing objects: 71% (46/64)
remote: Compressing objects: 73% (47/64)
remote: Compressing objects: 75% (48/64)
remote: Compressing objects: 76% (49/64)
remote: Compressing objects: 78% (50/64)
remote: Compressing objects: 79% (51/64)
remote: Compressing objects: 81% (52/64)
remote: Compressing objects: 82% (53/64)
remote: Compressing objects: 84% (54/64)
remote: Compressing objects: 85% (55/64)
remote: Compressing objects: 87% (56/64)
remote: Compressing objects: 89% (57/64)
remote: Compressing objects: 90% (58/64)
remote: Compressing objects: 92% (59/64)
remote: Compressing objects: 93% (60/64)
remote: Compressing objects: 95% (61/64)
remote: Compressing objects: 96% (62/64)
remote: Compressing objects: 98% (63/64)
remote: Compressing objects: 100% (64/64)
remote: Compressing objects: 100% (64/64), done.
.. parsed-literal::
Receiving objects: 0% (1/379)
.. parsed-literal::
Receiving objects: 1% (4/379)
.. parsed-literal::
Receiving objects: 2% (8/379)
Receiving objects: 3% (12/379)
Receiving objects: 4% (16/379)
Receiving objects: 5% (19/379)
Receiving objects: 6% (23/379)
Receiving objects: 7% (27/379)
Receiving objects: 8% (31/379)
Receiving objects: 9% (35/379)
Receiving objects: 10% (38/379)
Receiving objects: 11% (42/379)
Receiving objects: 12% (46/379)
Receiving objects: 13% (50/379)
Receiving objects: 14% (54/379)
Receiving objects: 15% (57/379)
Receiving objects: 16% (61/379)
Receiving objects: 17% (65/379)
Receiving objects: 18% (69/379)
Receiving objects: 19% (73/379)
Receiving objects: 20% (76/379)
Receiving objects: 21% (80/379)
Receiving objects: 22% (84/379)
Receiving objects: 23% (88/379)
Receiving objects: 24% (91/379)
.. parsed-literal::
Receiving objects: 25% (95/379)
.. parsed-literal::
Receiving objects: 26% (99/379)
.. parsed-literal::
Receiving objects: 27% (103/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 28% (107/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 29% (110/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 30% (114/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 31% (118/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 32% (122/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 33% (126/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 34% (129/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 35% (133/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 36% (137/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 37% (141/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 38% (145/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 39% (148/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 40% (152/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 41% (156/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 42% (160/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 43% (163/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 44% (167/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 45% (171/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 46% (175/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 47% (179/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 48% (182/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 49% (186/379), 9.38 MiB | 18.39 MiB/s
.. parsed-literal::
Receiving objects: 50% (190/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 51% (194/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 52% (198/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 53% (201/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 54% (205/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 55% (209/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 56% (213/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 57% (217/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 58% (220/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 59% (224/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 60% (228/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 61% (232/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 62% (235/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 63% (239/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 64% (243/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 65% (247/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 66% (251/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 67% (254/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 68% (258/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 69% (262/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 70% (266/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 71% (270/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 72% (273/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 73% (277/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 74% (281/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 75% (285/379), 9.38 MiB | 18.39 MiB/s
remote: Total 379 (delta 137), reused 113 (delta 113), pack-reused 202
Receiving objects: 76% (289/379), 9.38 MiB | 18.39 MiB/s
.. parsed-literal::
Receiving objects: 77% (292/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 78% (296/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 79% (300/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 80% (304/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 81% (307/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 82% (311/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 83% (315/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 84% (319/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 85% (323/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 86% (326/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 87% (330/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 88% (334/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 89% (338/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 90% (342/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 91% (345/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 92% (349/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 93% (353/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 94% (357/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 95% (361/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 96% (364/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 97% (368/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 98% (372/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 99% (376/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 100% (379/379), 9.38 MiB | 18.39 MiB/s
Receiving objects: 100% (379/379), 14.03 MiB | 19.52 MiB/s, done.
Resolving deltas: 0% (0/195)
Resolving deltas: 3% (7/195)
Resolving deltas: 6% (13/195)
Resolving deltas: 7% (15/195)
Resolving deltas: 8% (16/195)
Resolving deltas: 16% (32/195)
Resolving deltas: 17% (35/195)
Resolving deltas: 18% (37/195)
Resolving deltas: 19% (38/195)
Resolving deltas: 20% (39/195)
Resolving deltas: 21% (41/195)
Resolving deltas: 26% (51/195)
Resolving deltas: 41% (80/195)
Resolving deltas: 50% (99/195)
Resolving deltas: 52% (102/195)
Resolving deltas: 55% (109/195)
Resolving deltas: 57% (112/195)
Resolving deltas: 60% (117/195)
Resolving deltas: 61% (120/195)
Resolving deltas: 62% (121/195)
Resolving deltas: 68% (133/195)
Resolving deltas: 69% (135/195)
Resolving deltas: 73% (143/195)
Resolving deltas: 74% (145/195)
Resolving deltas: 75% (147/195)
Resolving deltas: 76% (149/195)
Resolving deltas: 78% (153/195)
Resolving deltas: 80% (157/195)
Resolving deltas: 81% (159/195)
Resolving deltas: 100% (195/195)
Resolving deltas: 100% (195/195), done.
.. parsed-literal::
remote: Counting objects: 100% (176/176), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 379 (delta 136), reused 111 (delta 111), pack-reused 203
Receiving objects: 100% (379/379), 14.03 MiB | 20.06 MiB/s, done.
Resolving deltas: 100% (195/195), done.
Cloning into 'EfficientSAM'...
.. parsed-literal::
remote: Enumerating objects: 424, done.
remote: Counting objects: 1% (1/85)
remote: Counting objects: 2% (2/85)
remote: Counting objects: 3% (3/85)
remote: Counting objects: 4% (4/85)
remote: Counting objects: 5% (5/85)
remote: Counting objects: 7% (6/85)
remote: Counting objects: 8% (7/85)
remote: Counting objects: 9% (8/85)
remote: Counting objects: 10% (9/85)
remote: Counting objects: 11% (10/85)
remote: Counting objects: 12% (11/85)
remote: Counting objects: 14% (12/85)
remote: Counting objects: 15% (13/85)
remote: Counting objects: 16% (14/85)
remote: Counting objects: 17% (15/85)
remote: Counting objects: 18% (16/85)
remote: Counting objects: 20% (17/85)
remote: Counting objects: 21% (18/85)
remote: Counting objects: 22% (19/85)
remote: Counting objects: 23% (20/85)
remote: Counting objects: 24% (21/85)
remote: Counting objects: 25% (22/85)
remote: Counting objects: 27% (23/85)
remote: Counting objects: 28% (24/85)
remote: Counting objects: 29% (25/85)
remote: Counting objects: 30% (26/85)
remote: Counting objects: 31% (27/85)
remote: Counting objects: 32% (28/85)
remote: Counting objects: 34% (29/85)
remote: Counting objects: 35% (30/85)
remote: Counting objects: 36% (31/85)
remote: Counting objects: 37% (32/85)
remote: Counting objects: 38% (33/85)
remote: Counting objects: 40% (34/85)
remote: Counting objects: 41% (35/85)
remote: Counting objects: 42% (36/85)
remote: Counting objects: 43% (37/85)
remote: Counting objects: 44% (38/85)
remote: Counting objects: 45% (39/85)
remote: Counting objects: 47% (40/85)
remote: Counting objects: 48% (41/85)
remote: Counting objects: 49% (42/85)
remote: Counting objects: 50% (43/85)
remote: Counting objects: 51% (44/85)
remote: Counting objects: 52% (45/85)
remote: Counting objects: 54% (46/85)
remote: Counting objects: 55% (47/85)
remote: Counting objects: 56% (48/85)
remote: Counting objects: 57% (49/85)
remote: Counting objects: 58% (50/85)
remote: Counting objects: 60% (51/85)
remote: Counting objects: 61% (52/85)
remote: Counting objects: 62% (53/85)
remote: Counting objects: 63% (54/85)
remote: Counting objects: 64% (55/85)
remote: Counting objects: 65% (56/85)
remote: Counting objects: 67% (57/85)
remote: Counting objects: 68% (58/85)
remote: Counting objects: 69% (59/85)
remote: Counting objects: 70% (60/85)
remote: Counting objects: 71% (61/85)
remote: Counting objects: 72% (62/85)
remote: Counting objects: 74% (63/85)
remote: Counting objects: 75% (64/85)
remote: Counting objects: 76% (65/85)
remote: Counting objects: 77% (66/85)
remote: Counting objects: 78% (67/85)
remote: Counting objects: 80% (68/85)
remote: Counting objects: 81% (69/85)
remote: Counting objects: 82% (70/85)
remote: Counting objects: 83% (71/85)
remote: Counting objects: 84% (72/85)
remote: Counting objects: 85% (73/85)
remote: Counting objects: 87% (74/85)
remote: Counting objects: 88% (75/85)
remote: Counting objects: 89% (76/85)
remote: Counting objects: 90% (77/85)
remote: Counting objects: 91% (78/85)
remote: Counting objects: 92% (79/85)
remote: Counting objects: 94% (80/85)
remote: Counting objects: 95% (81/85)
remote: Counting objects: 96% (82/85)
remote: Counting objects: 97% (83/85)
remote: Counting objects: 98% (84/85)
remote: Counting objects: 100% (85/85)
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 3% (1/33)
remote: Compressing objects: 6% (2/33)
remote: Compressing objects: 9% (3/33)
remote: Compressing objects: 12% (4/33)
remote: Compressing objects: 15% (5/33)
remote: Compressing objects: 18% (6/33)
remote: Compressing objects: 21% (7/33)
remote: Compressing objects: 24% (8/33)
remote: Compressing objects: 27% (9/33)
remote: Compressing objects: 30% (10/33)
remote: Compressing objects: 33% (11/33)
remote: Compressing objects: 36% (12/33)
remote: Compressing objects: 39% (13/33)
remote: Compressing objects: 42% (14/33)
remote: Compressing objects: 45% (15/33)
remote: Compressing objects: 48% (16/33)
remote: Compressing objects: 51% (17/33)
remote: Compressing objects: 54% (18/33)
remote: Compressing objects: 57% (19/33)
remote: Compressing objects: 60% (20/33)
remote: Compressing objects: 63% (21/33)
remote: Compressing objects: 66% (22/33)
remote: Compressing objects: 69% (23/33)
remote: Compressing objects: 72% (24/33)
remote: Compressing objects: 75% (25/33)
remote: Compressing objects: 78% (26/33)
remote: Compressing objects: 81% (27/33)
remote: Compressing objects: 84% (28/33)
remote: Compressing objects: 87% (29/33)
remote: Compressing objects: 90% (30/33)
remote: Compressing objects: 93% (31/33)
remote: Compressing objects: 96% (32/33)
remote: Compressing objects: 100% (33/33)
remote: Compressing objects: 100% (33/33), done.
.. parsed-literal::
Receiving objects: 0% (1/424)
Receiving objects: 1% (5/424)
.. parsed-literal::
Receiving objects: 2% (9/424)
Receiving objects: 3% (13/424)
Receiving objects: 4% (17/424)
Receiving objects: 5% (22/424)
.. parsed-literal::
Receiving objects: 5% (24/424), 17.34 MiB | 17.34 MiB/s
.. parsed-literal::
Receiving objects: 6% (26/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 7% (30/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 8% (34/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 9% (39/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 10% (43/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 11% (47/424), 29.54 MiB | 19.69 MiB/s
Receiving objects: 12% (51/424), 29.54 MiB | 19.69 MiB/s
.. parsed-literal::
Receiving objects: 12% (54/424), 41.72 MiB | 20.86 MiB/s
.. parsed-literal::
Receiving objects: 12% (54/424), 66.43 MiB | 22.08 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 66.43 MiB | 22.08 MiB/s
.. parsed-literal::
Receiving objects: 13% (56/424), 92.11 MiB | 22.90 MiB/s
.. parsed-literal::
Receiving objects: 14% (60/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 15% (64/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 16% (68/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 17% (73/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 18% (77/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 19% (81/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 20% (85/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 21% (90/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 22% (94/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 23% (98/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 24% (102/424), 105.12 MiB | 23.20 MiB/s
.. parsed-literal::
Receiving objects: 25% (106/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 26% (111/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 27% (115/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 28% (119/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 29% (123/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 30% (128/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 31% (132/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 32% (136/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 33% (140/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 34% (145/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 35% (149/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 36% (153/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 37% (157/424), 105.12 MiB | 23.20 MiB/s
Receiving objects: 38% (162/424), 105.12 MiB | 23.20 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 118.14 MiB | 24.69 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 144.35 MiB | 25.23 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 170.95 MiB | 25.59 MiB/s
.. parsed-literal::
Receiving objects: 38% (164/424), 197.61 MiB | 26.01 MiB/s
.. parsed-literal::
Receiving objects: 39% (166/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 40% (170/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 41% (174/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 42% (179/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 43% (183/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 44% (187/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 45% (191/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 46% (196/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 47% (200/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 48% (204/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 49% (208/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 50% (212/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 51% (217/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 52% (221/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 53% (225/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 54% (229/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 55% (234/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 56% (238/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 57% (242/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 58% (246/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 59% (251/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 60% (255/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 61% (259/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 62% (263/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 63% (268/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 64% (272/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 65% (276/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 66% (280/424), 197.61 MiB | 26.01 MiB/s
Receiving objects: 67% (285/424), 197.61 MiB | 26.01 MiB/s
.. parsed-literal::
Receiving objects: 67% (288/424), 222.04 MiB | 25.75 MiB/s
.. parsed-literal::
Receiving objects: 67% (288/424), 240.28 MiB | 24.07 MiB/s
.. parsed-literal::
Receiving objects: 68% (289/424), 240.28 MiB | 24.07 MiB/s
Receiving objects: 69% (293/424), 240.28 MiB | 24.07 MiB/s
Receiving objects: 70% (297/424), 240.28 MiB | 24.07 MiB/s
Receiving objects: 71% (302/424), 240.28 MiB | 24.07 MiB/s
Receiving objects: 72% (306/424), 240.28 MiB | 24.07 MiB/s
Receiving objects: 73% (310/424), 240.28 MiB | 24.07 MiB/s
.. parsed-literal::
Receiving objects: 73% (313/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 74% (314/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 75% (318/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 76% (323/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 77% (327/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 78% (331/424), 259.66 MiB | 22.62 MiB/s
.. parsed-literal::
Receiving objects: 79% (335/424), 259.66 MiB | 22.62 MiB/s
.. parsed-literal::
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 424 (delta 76), reused 52 (delta 52), pack-reused 339
Receiving objects: 80% (340/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 81% (344/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 82% (348/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 83% (352/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 84% (357/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 85% (361/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 86% (365/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 87% (369/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 88% (374/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 89% (378/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 90% (382/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 91% (386/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 92% (391/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 93% (395/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 94% (399/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 95% (403/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 96% (408/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 97% (412/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 98% (416/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 99% (420/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 100% (424/424), 259.66 MiB | 22.62 MiB/s
Receiving objects: 100% (424/424), 262.14 MiB | 23.41 MiB/s, done.
Resolving deltas: 0% (0/246)
Resolving deltas: 4% (10/246)
Resolving deltas: 6% (17/246)
Resolving deltas: 15% (37/246)
Resolving deltas: 18% (46/246)
Resolving deltas: 22% (56/246)
Resolving deltas: 23% (57/246)
Resolving deltas: 24% (60/246)
Resolving deltas: 26% (64/246)
Resolving deltas: 32% (81/246)
Resolving deltas: 36% (90/246)
Resolving deltas: 37% (92/246)
Resolving deltas: 38% (94/246)
Resolving deltas: 41% (101/246)
.. parsed-literal::
Resolving deltas: 43% (108/246)
Resolving deltas: 45% (112/246)
.. parsed-literal::
Resolving deltas: 48% (119/246)
Resolving deltas: 49% (121/246)
Resolving deltas: 51% (127/246)
Resolving deltas: 52% (128/246)
Resolving deltas: 54% (133/246)
Resolving deltas: 57% (142/246)
Resolving deltas: 61% (152/246)
Resolving deltas: 62% (154/246)
Resolving deltas: 65% (162/246)
Resolving deltas: 66% (164/246)
Resolving deltas: 67% (165/246)
Resolving deltas: 69% (172/246)
Resolving deltas: 70% (174/246)
Resolving deltas: 88% (217/246)
Resolving deltas: 96% (237/246)
Resolving deltas: 97% (240/246)
Resolving deltas: 98% (243/246)
.. parsed-literal::
Resolving deltas: 99% (245/246)
.. parsed-literal::
Resolving deltas: 100% (246/246)
Resolving deltas: 100% (246/246), done.
Receiving objects: 100% (424/424), 262.14 MiB | 28.00 MiB/s, done.
Resolving deltas: 100% (246/246), done.
.. code:: ipython3
@ -927,7 +244,7 @@ GroundingDINO imports
.. parsed-literal::
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3549.)
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3587.)
.. parsed-literal::
@ -935,6 +252,11 @@ GroundingDINO imports
final text_encoder_type: bert-base-uncased
.. parsed-literal::
FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
.. parsed-literal::
final text_encoder_type: bert-base-uncased
@ -1018,18 +340,10 @@ Convert GroundingDINO to OpenVINO IR format
TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
.. parsed-literal::
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
.. parsed-literal::
TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
@ -1040,16 +354,30 @@ Convert GroundingDINO to OpenVINO IR format
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
.. parsed-literal::
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
.. parsed-literal::
output layer_id 0 is nan
num_nan 230400, num_inf 0
output layer_id 1 is nan
num_nan 230400, num_inf 0
output layer_id 2 is nan
num_nan 230400, num_inf 0
output layer_id 3 is nan
num_nan 230400, num_inf 0
output layer_id 4 is nan
num_nan 230400, num_inf 0
output layer_id 5 is nan
num_nan 230400, num_inf 0
Run OpenVINO optimized GroundingDINO
@ -1199,14 +527,10 @@ class, but the inference will be done using OpenVINO optimized model.
.. parsed-literal::
2024-04-17 23:56:30.569255: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:56:30.608409: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:14:36.448862: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:14:36.488990: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:56:31.167736: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-07 00:14:37.051985: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Convert predicted boxes to supervision box detections format
@ -1232,7 +556,7 @@ Draw box detections
.. parsed-literal::
SupervisionWarnings: BoxAnnotator is deprecated: `BoxAnnotator` is deprecated and will be removed in `supervision-0.22.0`. Use `BoundingBoxAnnotator` and `LabelAnnotator` instead
SupervisionWarnings: annotate is deprecated: `BoxAnnotator` is deprecated and will be removed in `supervision-0.22.0`. Use `BoundingBoxAnnotator` and `LabelAnnotator` instead
@ -1285,10 +609,6 @@ segmentation. First of all let’s convert ``SAM`` model to OpenVINO IR.
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
.. parsed-literal::
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
@ -1491,7 +811,7 @@ Combine both boxes and segmentation masks and draw them.
.. parsed-literal::
SupervisionWarnings: BoxAnnotator is deprecated: `BoxAnnotator` is deprecated and will be removed in `supervision-0.22.0`. Use `BoundingBoxAnnotator` and `LabelAnnotator` instead
SupervisionWarnings: annotate is deprecated: `BoxAnnotator` is deprecated and will be removed in `supervision-0.22.0`. Use `BoundingBoxAnnotator` and `LabelAnnotator` instead

4
docs/notebooks/handwritten-ocr-with-output.rst
View File

@ -52,10 +52,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.

969
docs/notebooks/hello-npu-with-output.rst Normal file
View File

@ -0,0 +1,969 @@
Hello NPU
=========
Working with NPU in OpenVINO™
-----------------------------
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Introduction <#introduction>`__
- `Install required packages <#install-required-packages>`__
- `Checking NPU with Query Device <#checking-npu-with-query-device>`__
- `List the NPU with
core.available_devices <#list-the-npu-with-core-available_devices>`__
- `Check Properties with
core.get_property <#check-properties-with-core-get_property>`__
- `Brief Descriptions of Key
Properties <#brief-descriptions-of-key-properties>`__
- `Compiling a Model on NPU <#compiling-a-model-on-npu>`__
- `Download and Convert a Model <#download-and-convert-a-model>`__
- `Download the Model <#download-the-model>`__
- `Convert the Model to OpenVINO IR
format <#convert-the-model-to-openvino-ir-format>`__
- `Compile with Default
Configuration <#compile-with-default-configuration>`__
- `Reduce Compile Time through Model
Caching <#reduce-compile-time-through-model-caching>`__
- `UMD Model Caching <#umd-model-caching>`__
- `OpenVINO Model Caching <#openvino-model-caching>`__
- `Throughput and Latency Performance
Hints <#throughput-and-latency-performance-hints>`__
- `Performance Comparison with
benchmark_app <#performance-comparison-with-benchmark_app>`__
- `NPU vs CPU with Latency Hint <#npu-vs-cpu-with-latency-hint>`__
- `Effects of UMD Model
Caching <#effects-of-umd-model-caching>`__
- `NPU vs CPU with Throughput
Hint <#npu-vs-cpu-with-throughput-hint>`__
- `Limitations <#limitations>`__
- `Conclusion <#conclusion>`__
This tutorial provides a high-level overview of working with the NPU
device **Intel(R) AI Boost** (introduced with the Intel® Core™ Ultra
generation of CPUs) in OpenVINO. It explains some of the key properties
of the NPU and shows how to compile a model on NPU with performance
hints.
This tutorial also shows example commands for benchmark_app that can be
run to compare NPU performance with CPU in different configurations.
Introduction
------------
The Neural Processing Unit (NPU) is a low power hardware solution which
enables you to offload certain neural network computation tasks from
other devices, for more streamlined resource management.
Note that the NPU plugin is included in PIP installation of OpenVINO™
and you need to `install a proper NPU
driver <https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-npu.html>`__
to use it successfully.
| **Supported Platforms**:
| Host: Intel® Core™ Ultra
| NPU device: NPU 3720
| OS: Ubuntu 22.04 (with Linux Kernel 6.6+), MS Windows 11 (both 64-bit)
To learn more about the NPU Device, see the
`page <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html>`__.
Install required packages
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%pip install -q "openvino>=2024.1.0" torch torchvision --extra-index-url https://download.pytorch.org/whl/cpu
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
Checking NPU with Query Device
------------------------------
In this section, we will see how to list the available NPU and check its
properties. Some of the key properties will be defined.
List the NPU with core.available_devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO Runtime provides the ``available_devices`` method for checking
which devices are available for inference. The following code will
output a list a compatible OpenVINO devices, in which Intel NPU should
appear (ensure that the driver is installed successfully).
.. code:: ipython3
import openvino as ov
core = ov.Core()
core.available_devices
.. parsed-literal::
['CPU', 'GPU', 'NPU']
Check Properties with core.get_property
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get information about the NPU, we can use device properties. In
OpenVINO, devices have properties that describe their characteristics
and configurations. Each property has a name and associated value that
can be queried with the ``get_property`` method.
To get the value of a property, such as the device name, we can use the
``get_property`` method as follows:
.. code:: ipython3
device = "NPU"
core.get_property(device, "FULL_DEVICE_NAME")
.. parsed-literal::
'Intel(R) AI Boost'
Each device also has a specific property called
``SUPPORTED_PROPERTIES``, that enables viewing all the available
properties in the device. We can check the value for each property by
simply looping through the dictionary returned by
``core.get_property("NPU", "SUPPORTED_PROPERTIES")`` and then querying
for that property.
.. code:: ipython3
print(f"{device} SUPPORTED_PROPERTIES:\n")
supported_properties = core.get_property(device, "SUPPORTED_PROPERTIES")
indent = len(max(supported_properties, key=len))
for property_key in supported_properties:
if property_key not in ("SUPPORTED_METRICS", "SUPPORTED_CONFIG_KEYS", "SUPPORTED_PROPERTIES"):
try:
property_val = core.get_property(device, property_key)
except TypeError:
property_val = "UNSUPPORTED TYPE"
print(f"{property_key:<{indent}}: {property_val}")
Brief Descriptions of Key Properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each device has several properties as seen in the last command. Some of
the key properties are: - ``FULL_DEVICE_NAME`` - The product name of the
NPU. - ``PERFORMANCE_HINT`` - A high-level way to tune the device for a
specific performance metric, such as latency or throughput, without
worrying about device-specific settings. - ``CACHE_DIR`` - The directory
where the OpenVINO model cache data is stored to speed up the
compilation time. - ``OPTIMIZATION_CAPABILITIES`` - The model data types
(INT8, FP16, FP32, etc) that are supported by this NPU.
To learn more about devices and properties, see the `Query Device
Properties <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/query-device-properties.html>`__
page.
Compiling a Model on NPU
------------------------
Now, we know the NPU present in the system and we have checked its
properties. We can easily use it for compiling and running models with
OpenVINO NPU plugin.
Download and Convert a Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This tutorial uses the ``resnet50`` model. The ``resnet50`` model is
used for image classification tasks. The model was trained on
`ImageNet <https://www.image-net.org/index.php>`__ dataset which
contains over a million images categorized into 1000 classes. To read
more about resnet50, see the
`paper <https://ieeexplore.ieee.org/document/7780459>`__.
Download the Model
^^^^^^^^^^^^^^^^^^
Fetch `ResNet50
CV <https://pytorch.org/vision/stable/models/generated/torchvision.models.resnet50.html#torchvision.models.ResNet50_Weights>`__
Classification model from torchvision.
.. code:: ipython3
from pathlib import Path
# create a directory for resnet model file
MODEL_DIRECTORY_PATH = Path("model")
MODEL_DIRECTORY_PATH.mkdir(exist_ok=True)
model_name = "resnet50"
.. code:: ipython3
from torchvision.models import resnet50, ResNet50_Weights
# create model object
pytorch_model = resnet50(weights=ResNet50_Weights.DEFAULT)
# switch model from training to inference mode
pytorch_model.eval();
Convert the Model to OpenVINO IR format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To convert this Pytorch model to OpenVINO IR with ``FP16`` precision,
use model conversion API. The models are saved to the
``model/ir_model/`` directory. For more details about model conversion,
see this
`page <https://docs.openvino.ai/2024/openvino-workflow/model-preparation.html>`__.
.. code:: ipython3
precision = "FP16"
model_path = MODEL_DIRECTORY_PATH / "ir_model" / f"{model_name}_{precision.lower()}.xml"
model = None
if not model_path.exists():
model = ov.convert_model(pytorch_model, input=[[1, 3, 224, 224]])
ov.save_model(model, model_path, compress_to_fp16=(precision == "FP16"))
print("IR model saved to {}".format(model_path))
else:
print("Read IR model from {}".format(model_path))
model = core.read_model(model_path)
.. parsed-literal::
Read IR model from model\ir_model\resnet50_fp16.xml
**Note:** NPU also supports ``INT8`` quantized models.
Compile with Default Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When the model is ready, first we need to read it, using the
``read_model`` method. Then, we can use the ``compile_model`` method and
specify the name of the device we want to compile the model on, in this
case, “NPU”.
.. code:: ipython3
compiled_model = core.compile_model(model, device)
Reduce Compile Time through Model Caching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Depending on the model used, device-specific optimizations and network
compilations can cause the compile step to be time-consuming, especially
with larger models, which may lead to bad user experience in the
application. To solve this **Model Caching** can be used.
Model Caching helps reduce application startup delays by exporting and
reusing the compiled model automatically. The following two
compilation-related metrics are crucial in this area:
- **First-Ever Inference Latency (FEIL)**:
Measures all steps required to compile and execute a model on the
device for the first time. It includes model compilation time, the
time required to load and initialize the model on the device and the
first inference execution.
- **First Inference Latency (FIL)**:
Measures the time required to load and initialize the pre-compiled
model on the device and the first inference execution.
In NPU, UMD model caching is a solution enabled by default by the
driver. It improves time to first inference (FIL) by storing the model
in the cache after compilation (included in FEIL). Learn more about UMD
Caching
`here <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html#umd-dynamic-model-caching>`__.
Due to this caching, it takes lesser time to load the model after first
compilation.
| You can also use OpenVINO Model Caching, which is a common mechanism
for all OpenVINO device plugins and can be enabled by setting the
``cache_dir`` property.
| By enabling OpenVINO Model Caching, the UMD caching is automatically
bypassed by the NPU plugin, which means the model will only be stored
in the OpenVINO cache after compilation. When a cache hit occurs for
subsequent compilation requests, the plugin will import the model
instead of recompiling it.
UMD Model Caching
^^^^^^^^^^^^^^^^^
To see how UMD caching see the following example:
.. code:: ipython3
import time
from pathlib import Path
start = time.time()
core = ov.Core()
# Compile the model as before
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"UMD Caching (first time) - compile time: {time.time() - start}s")
.. parsed-literal::
UMD Caching (first time) - compile time: 3.2854952812194824s
.. code:: ipython3
start = time.time()
core = ov.Core()
# Compile the model once again to see UMD Caching
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"UMD Caching - compile time: {time.time() - start}s")
.. parsed-literal::
UMD Caching - compile time: 2.269814968109131s
OpenVINO Model Caching
^^^^^^^^^^^^^^^^^^^^^^
To get an idea of OpenVINO model caching, we can use the OpenVINO cache
as follow
.. code:: ipython3
# Create cache folder
cache_folder = Path("cache")
cache_folder.mkdir(exist_ok=True)
start = time.time()
core = ov.Core()
# Set cache folder
core.set_property({"CACHE_DIR": cache_folder})
# Compile the model
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"Cache enabled (first time) - compile time: {time.time() - start}s")
start = time.time()
core = ov.Core()
# Set cache folder
core.set_property({"CACHE_DIR": cache_folder})
# Compile the model as before
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"Cache enabled (second time) - compile time: {time.time() - start}s")
.. parsed-literal::
Cache enabled (first time) - compile time: 0.6362860202789307s
Cache enabled (second time) - compile time: 0.3032548427581787s
And when the OpenVINO cache is disabled:
.. code:: ipython3
start = time.time()
core = ov.Core()
model = core.read_model(model=model_path)
compiled_model = core.compile_model(model, device)
print(f"Cache disabled - compile time: {time.time() - start}s")
.. parsed-literal::
Cache disabled - compile time: 3.0127954483032227s
The actual time improvements will depend on the environment as well as
the model being used but it is definitely something to consider when
optimizing an application. To read more about this, see the `Model
Caching
docs <https://docs.openvino.ai/2024/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html>`__.
Throughput and Latency Performance Hints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To simplify device and pipeline configuration, OpenVINO provides
high-level performance hints that automatically set the batch size and
number of parallel threads for inference. The “LATENCY” performance hint
optimizes for fast inference times while the “THROUGHPUT” performance
hint optimizes for high overall bandwidth or FPS.
To use the “LATENCY” performance hint, add
``{"PERFORMANCE_HINT": "LATENCY"}`` when compiling the model as shown
below. For NPU, this automatically minimizes the batch size and number
of parallel streams such that all of the compute resources can focus on
completing a single inference as fast as possible.
.. code:: ipython3
compiled_model = core.compile_model(model, device, {"PERFORMANCE_HINT": "LATENCY"})
To use the “THROUGHPUT” performance hint, add
``{"PERFORMANCE_HINT": "THROUGHPUT"}`` when compiling the model. For
NPUs, this creates multiple processing streams to efficiently utilize
all the execution cores and optimizes the batch size to fill the
available memory.
.. code:: ipython3
compiled_model = core.compile_model(model, device, {"PERFORMANCE_HINT": "THROUGHPUT"})
Performance Comparison with benchmark_app
-----------------------------------------
Given all the different options available when compiling a model, it may
be difficult to know which settings work best for a certain application.
Thankfully, OpenVINO provides ``benchmark_app`` - a performance
benchmarking tool.
The basic syntax of ``benchmark_app`` is as follows:
``benchmark_app -m PATH_TO_MODEL -d TARGET_DEVICE -hint {throughput,cumulative_throughput,latency,none}``
where ``TARGET_DEVICE`` is any device shown by the ``available_devices``
method as well as the MULTI and AUTO devices we saw previously, and the
value of hint should be one of the values between brackets.
Note that benchmark_app only requires the model path to run but both
device and hint arguments will be useful to us. For more advanced
usages, the tool itself has other options that can be checked by running
``benchmark_app -h`` or reading the
`docs <https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-tool.html>`__.
The following example shows us to benchmark a simple model, using a NPU
with latency focus:
``benchmark_app -m {model_path} -d NPU -hint latency``
| For completeness, let us list here some of the comparisons we may want
to do by varying the device and hint used. Note that the actual
performance may depend on the hardware used. Generally, we should
expect NPU to be better than CPU.
| Please refer to the ``benchmark_app`` log entries under
``[Step 11/11] Dumping statistics report`` to observe the differences
in latency and throughput between the CPU and NPU..
NPU vs CPU with Latency Hint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
!benchmark_app -m {model_path} -d CPU -hint latency
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 14.00 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 143.22 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model2
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] NUM_STREAMS: 1
[ INFO ] AFFINITY: Affinity.HYBRID_AWARE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: False
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: False
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 28.95 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 1612 iterations
[ INFO ] Duration: 60039.72 ms
[ INFO ] Latency:
[ INFO ] Median: 39.99 ms
[ INFO ] Average: 37.13 ms
[ INFO ] Min: 19.13 ms
[ INFO ] Max: 71.94 ms
[ INFO ] Throughput: 26.85 FPS
.. code:: ipython3
!benchmark_app -m {model_path} -d NPU -hint latency
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ] Device info:
[ INFO ] NPU
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 11.51 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 2302.40 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] DEVICE_ID:
[ INFO ] ENABLE_CPU_PINNING: False
[ INFO ] EXECUTION_DEVICES: NPU.3720
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float16'>
[ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] NETWORK_NAME:
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.94 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:NPU.3720
[ INFO ] Count: 17908 iterations
[ INFO ] Duration: 60004.49 ms
[ INFO ] Latency:
[ INFO ] Median: 3.29 ms
[ INFO ] Average: 3.33 ms
[ INFO ] Min: 3.21 ms
[ INFO ] Max: 6.90 ms
[ INFO ] Throughput: 298.44 FPS
Effects of UMD Model Caching
''''''''''''''''''''''''''''
To see the effects of UMD Model caching, we are going to run the
benchmark_app and see the difference in model read time and compilation
time:
.. code:: ipython3
!benchmark_app -m {model_path} -d NPU -hint latency
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ] Device info:
[ INFO ] NPU
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 11.00 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 2157.58 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] DEVICE_ID:
[ INFO ] ENABLE_CPU_PINNING: False
[ INFO ] EXECUTION_DEVICES: NPU.3720
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float16'>
[ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] NETWORK_NAME:
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.94 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:NPU.3720
[ INFO ] Count: 17894 iterations
[ INFO ] Duration: 60004.76 ms
[ INFO ] Latency:
[ INFO ] Median: 3.29 ms
[ INFO ] Average: 3.33 ms
[ INFO ] Min: 3.21 ms
[ INFO ] Max: 14.38 ms
[ INFO ] Throughput: 298.21 FPS
As you can see from the log entries ``[Step 4/11] Reading model files``
and ``[Step 7/11] Loading the model to the device``, it takes less time
to read and compile the model after the initial load.
NPU vs CPU with Throughput Hint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
!benchmark_app -m {model_path} -d CPU -hint throughput
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 12.00 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 177.18 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model2
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ] NUM_STREAMS: 4
[ INFO ] AFFINITY: Affinity.HYBRID_AWARE
[ INFO ] INFERENCE_NUM_THREADS: 16
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: False
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 31.62 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 3212 iterations
[ INFO ] Duration: 60082.26 ms
[ INFO ] Latency:
[ INFO ] Median: 65.28 ms
[ INFO ] Average: 74.60 ms
[ INFO ] Min: 35.65 ms
[ INFO ] Max: 157.31 ms
[ INFO ] Throughput: 53.46 FPS
.. code:: ipython3
!benchmark_app -m {model_path} -d NPU -hint throughput
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ] Device info:
[ INFO ] NPU
[ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 11.50 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 2265.07 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] DEVICE_ID:
[ INFO ] ENABLE_CPU_PINNING: False
[ INFO ] EXECUTION_DEVICES: NPU.3720
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float16'>
[ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] NETWORK_NAME:
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
[ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.95 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:NPU.3720
[ INFO ] Count: 19080 iterations
[ INFO ] Duration: 60024.79 ms
[ INFO ] Latency:
[ INFO ] Median: 12.51 ms
[ INFO ] Average: 12.56 ms
[ INFO ] Min: 6.92 ms
[ INFO ] Max: 25.80 ms
[ INFO ] Throughput: 317.87 FPS
Limitations
-----------
1. Currently, only the models with static shapes are supported on NPU.
2. If the path to the model file includes non-Unicode symbols, such as
in Chinese, the model cannot be used for inference on NPU. It will
return an error.
Conclusion
----------
This tutorial demonstrates how easy it is to use NPU in OpenVINO, check
its properties, and even tailor the model performance through the
different performance hints.
Discover the power of Neural Processing Unit (NPU) with OpenVINO through
these interactive Jupyter notebooks: ##### Introduction -
`hello-world <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/hello-world>`__:
Start your OpenVINO journey by performing inference on an OpenVINO IR
model. -
`hello-segmentation <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/hello-segmentation>`__:
Dive into inference with a segmentation model and explore image
segmentation capabilities.
Model Optimization and Conversion
'''''''''''''''''''''''''''''''''
- `model-tools <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/model-tools>`__:
Discover how to download, convert, and benchmark models from the Open
Model Zoo.
- `tflite-to-openvino <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/tflite-to-openvino>`__:
Learn the process of converting TensorFlow Lite models to OpenVINO IR
format.
- `yolov7-optimization <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov7-optimization>`__:
Optimize the YOLOv7 model for enhanced performance in OpenVINO.
- `yolov8-optimization <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov8-optimization>`__:
Convert and optimize YOLOv8 models for efficient deployment with
OpenVINO.
Advanced Computer Vision Techniques
'''''''''''''''''''''''''''''''''''
- `vision-background-removal <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-background-removal>`__:
Implement advanced image segmentation and background manipulation
with U^2-Net.
- `handwritten-ocr <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/handwritten-ocr>`__:
Apply optical character recognition to handwritten Chinese and
Japanese text.
- `image-inpainting <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/image-inpainting>`__:
Explore the art of image in-painting and restore images with missing
parts.
- `vehicle-detection-and-recognition <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vehicle-detection-and-recognition>`__:
Use pre-trained models for vehicle detection and recognition in
images.
- `vision-image-colorization <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-image-colorization>`__:
Bring black and white images to life by adding color with neural
networks.
Real-Time Webcam Applications
'''''''''''''''''''''''''''''
- `tflite-selfie-segmentation <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/tflite-selfie-segmentation>`__:
Apply TensorFlow Lite models for selfie segmentation and background
processing.
- `object-detection-webcam <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/object-detection-webcam>`__:
Experience real-time object detection using your webcam and OpenVINO.
- `pose-estimation-webcam <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pose-estimation-webcam>`__:
Perform human pose estimation in real-time with webcam integration.
- `action-recognition-webcam <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/action-recognition-webcam>`__:
Recognize and classify human actions live with your webcam.
- `style-transfer-webcam <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/style-transfer-webcam>`__:
Transform your webcam feed with artistic styles in real-time using
pre-trained models.
- `3D-pose-estimation-webcam <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pose-estimation-webcam>`__:
Perform 3D multi-person pose estimation with OpenVINO.

8
docs/notebooks/hello-segmentation-with-output.rst
View File

@ -38,10 +38,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -198,7 +194,7 @@ is provided.
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fcde9549940>
<matplotlib.image.AxesImage at 0x7f34a03f2a60>
@ -225,7 +221,7 @@ Do Inference
.. parsed-literal::
<matplotlib.image.AxesImage at 0x7fcd9050b400>
<matplotlib.image.AxesImage at 0x7f34a0321cd0>

4
docs/notebooks/hello-world-with-output.rst
View File

@ -42,10 +42,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.

149
docs/notebooks/hugging-face-hub-with-output.rst
View File

@ -73,15 +73,7 @@ Installing Requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -129,6 +121,8 @@ tutorials <https://huggingface.co/learn/nlp-course/chapter2/2?fw=pt#behind-the-p
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
@ -193,7 +187,7 @@ Note how we reuse our real ``encoded_input``, passing it to the
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
@ -331,23 +325,11 @@ documentation <https://huggingface.co/docs/optimum/intel/inference>`__.
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
.. parsed-literal::
2024-04-17 23:58:07.506496: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:58:07.542304: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:16:16.370934: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:16:16.406467: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:58:08.114190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
2024-05-07 00:16:17.026247: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
@ -384,28 +366,14 @@ inference run.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Framework not specified. Using pt to export the model.
.. parsed-literal::
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
.. parsed-literal::
Using framework PyTorch: 2.3.0+cpu
Overriding 1 configuration item(s)
.. parsed-literal::
- use_cache -> False
@ -416,12 +384,8 @@ inference run.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
.. parsed-literal::
Compiling the model to AUTO ...
@ -476,11 +440,7 @@ Full list of supported arguments available via ``--help``
.. parsed-literal::
2024-04-17 23:58:19.217675: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-05-07 00:16:28.374572: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
usage: optimum-cli export openvino [-h] -m MODEL [--task TASK]
[--cache_dir CACHE_DIR]
[--framework {pt,tf}] [--trust-remote-code]
@ -489,7 +449,9 @@ Full list of supported arguments available via ``--help``
[--weight-format {fp32,fp16,int8,int4,int4_sym_g128,int4_asym_g128,int4_sym_g64,int4_asym_g64}]
[--ratio RATIO] [--sym]
[--group-size GROUP_SIZE]
[--disable-stateful] [--convert-tokenizer]
[--dataset DATASET] [--disable-stateful]
[--disable-convert-tokenizer]
[--convert-tokenizer]
[--library {transformers,diffusers,timm,sentence_transformers}]
output
@ -507,21 +469,21 @@ Full list of supported arguments available via ``--help``
--task TASK The task to export the model for. If not specified,
the task will be auto-inferred based on the model.
Available tasks depend on the model, but are among:
['masked-im', 'fill-mask', 'token-classification',
'conversational', 'image-segmentation', 'zero-shot-
image-classification', 'audio-frame-classification',
'audio-classification', 'feature-extraction', 'zero-
shot-object-detection', 'text2text-generation',
'automatic-speech-recognition', 'text-generation',
'image-to-text', 'semantic-segmentation', 'text-
classification', 'sentence-similarity', 'audio-
xvector', 'depth-estimation', 'object-detection',
'stable-diffusion', 'image-classification', 'mask-
generation', 'multiple-choice', 'stable-diffusion-xl',
'image-to-image', 'text-to-audio', 'question-
answering']. For decoder models, use `xxx-with-past`
to export the model using past key values in the
decoder.
['feature-extraction', 'zero-shot-image-
classification', 'conversational', 'text2text-
generation', 'sentence-similarity', 'stable-
diffusion', 'text-to-audio', 'stable-diffusion-xl',
'fill-mask', 'image-to-text', 'text-generation',
'semantic-segmentation', 'automatic-speech-
recognition', 'mask-generation', 'token-
classification', 'audio-classification', 'multiple-
choice', 'question-answering', 'masked-im', 'zero-
shot-object-detection', 'audio-xvector', 'image-
segmentation', 'object-detection', 'image-
classification', 'audio-frame-classification', 'image-
to-image', 'depth-estimation', 'text-classification'].
For decoder models, use `xxx-with-past` to export the
model using past key values in the decoder.
--cache_dir CACHE_DIR
Path indicating where to store cache.
--framework {pt,tf} The framework to use for the export. If not provided,
@ -552,6 +514,13 @@ Full list of supported arguments available via ``--help``
--group-size GROUP_SIZE
The group size to use for quantization. Recommended
value is 128 and -1 uses per-column quantization.
--dataset DATASET The dataset used for data-aware compression or
quantization with NNCF. You can use the one from the
list ['wikitext2','c4','c4-new','ptb','ptb-new'] for
LLLMs or
['conceptual_captions','laion/220k-GPT4Vision-
captions-from-LIVIS','laion/filtered-wit'] for
diffusion models.
--disable-stateful Disable stateful converted models, stateless models
will be generated instead. Stateful models are
produced by default when this key is not used. In
@ -563,8 +532,11 @@ Full list of supported arguments available via ``--help``
a stateless model, for example, to be compatible with
existing OpenVINO native inference code that expects
kv-cache inputs and outputs in the model.
--convert-tokenizer Add converted tokenizer and detokenizer with OpenVINO
Tokenizers
--disable-convert-tokenizer
Do not add converted tokenizer and detokenizer
OpenVINO models.
--convert-tokenizer [Deprecated] Add converted tokenizer and detokenizer
with OpenVINO Tokenizers.
--library {transformers,diffusers,timm,sentence_transformers}
The library on the model. If not provided, will
attempt to infer the local checkpoint's library
@ -588,44 +560,25 @@ compression:
.. parsed-literal::
2024-04-17 23:58:23.821792: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-05-07 00:16:33.077270: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
.. parsed-literal::
`--fp16` option is deprecated and will be removed in a future version. Use `--weight-format` instead.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Framework not specified. Using pt to export the model.
.. parsed-literal::
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
Using framework PyTorch: 2.3.0+cpu
Overriding 1 configuration item(s)
- use_cache -> False
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
OpenVINO Tokenizers is not available. To deploy models in production with C++ code, please follow installation instructions: https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#installation
Tokenizer won't be converted.
After export, model will be available in the specified directory and can

872
docs/notebooks/image-classification-quantization-with-output.rst
View File

File diff suppressed because it is too large Load Diff

0
docs/notebooks/image-classification-quantization-with-output_files/image-classification-quantization-with-output_30_5.png → docs/notebooks/image-classification-quantization-with-output_files/image-classification-quantization-with-output_30_2.png
View File

2
docs/notebooks/instant-id-with-output.rst
View File

@ -645,7 +645,7 @@ LoRA <https://huggingface.co/latent-consistency/lcm-lora-sdxl>`__ to
speedup the generation process. Previously, we already considered how to
convert and run SDXL model for Text-to-Image and Image-to-Image
generation using Optimum-Intel library (please check out this notebook
for `details <stable-diffusion-xl-with-output.html>`__ ), now
for `details <stable-diffusion-xl-with-output.html>`__), now
we will use it in combination with ControlNet and convert it using
OpenVINO Model Conversion API.

45
docs/notebooks/knowledge-graphs-conve-with-output.rst
View File

@ -222,7 +222,7 @@ Download Model Checkpoint
.. parsed-literal::
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/knowledge-graphs-conve/models/conve.pt')
PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/knowledge-graphs-conve/models/conve.pt')
@ -384,7 +384,7 @@ typical to use metrics such as Mean Reciprocal Rank, Hits@10 etc.
.. parsed-literal::
Average time taken for inference: 0.6107787291208903 ms
Average time taken for inference: 0.7582604885101318 ms
Mean accuracy of the model on the test dataset: 0.875
@ -531,7 +531,7 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
Average time taken for inference: 0.6598830223083496 ms
Average time taken for inference: 0.675062338511149 ms
Mean accuracy of the model on the test dataset: 0.10416666666666667
@ -550,7 +550,7 @@ Determine the platform specific speedup obtained through OpenVINO graph optimiza
.. parsed-literal::
Speedup with OpenVINO optimizations: 0.93 X
Speedup with OpenVINO optimizations: 1.12 X
Benchmark the converted OpenVINO model using benchmark app
@ -580,30 +580,22 @@ inference can also be obtained by looking at the benchmark app results.
.. parsed-literal::
Benchmark OpenVINO model using the benchmark app
.. parsed-literal::
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
.. parsed-literal::
[ INFO ] Read model took 13.96 ms
[ INFO ] Read model took 4.87 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] e1 (node: e1) : i64 / [...] / []
@ -619,11 +611,7 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: aten::softmax/Softmax) : f32 / [...] / [1,271]
[Step 7/11] Loading the model to the device
.. parsed-literal::
[ INFO ] Compile model took 82.08 ms
[ INFO ] Compile model took 69.30 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@ -638,6 +626,7 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
@ -652,21 +641,17 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] Fill input 'rel' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 10000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 1.43 ms
.. parsed-literal::
[ INFO ] First inference took 1.25 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 101412 iterations
[ INFO ] Duration: 10001.13 ms
[ INFO ] Count: 101688 iterations
[ INFO ] Duration: 10000.88 ms
[ INFO ] Latency:
[ INFO ] Median: 1.01 ms
[ INFO ] Average: 1.02 ms
[ INFO ] Min: 0.60 ms
[ INFO ] Max: 8.57 ms
[ INFO ] Throughput: 10140.06 FPS
[ INFO ] Min: 0.70 ms
[ INFO ] Max: 8.75 ms
[ INFO ] Throughput: 10167.91 FPS
Conclusions

69
docs/notebooks/kosmos2-multimodal-large-language-model-with-output.rst
View File

@ -66,31 +66,10 @@ Install requirements
.. parsed-literal::
Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
.. parsed-literal::
Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -154,18 +133,14 @@ example <https://huggingface.co/microsoft/kosmos-2-patch14-224>`__
.. parsed-literal::
2024-04-18 00:01:02.253505: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-18 00:01:02.287838: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:19:14.165808: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:19:14.200484: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-18 00:01:02.846922: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
2024-05-07 00:19:14.695568: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
@ -385,19 +360,11 @@ Vision model accept ``pixel_values`` and returns ``image_embeds``.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:471: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:469: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:511: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:509: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
@ -425,7 +392,7 @@ Convert Image To Text Projection model
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:165: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:165: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)
if a.grad is not None:
@ -560,17 +527,13 @@ generated text by ``AutoProcessor``.
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:810: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:808: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if max_pos > self.weights.size(0):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1119: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1117: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if input_shape[-1] > 1:
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:926: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:924: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attention_mask.size() != (batch_size, 1, seq_length, src_len):
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1212: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1210: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if past_key_values_length > 0:

4
docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c0a83eb915195973a7315333a239ea60abf597988effc7fcca7dfea302afb625
size 117636
oid sha256:9ab062bc4b0b22c2815ec0f5c3a2e7237c9ae71ac83cc94e3363de499d8a03c6
size 118458

4
docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8d18af510a639c26185fec562aed50889b07e16f73465e7cd071b97634995d78
size 1151036
oid sha256:a3943e3c5bcdf600ff01c05b9f60dc88b9bc7bd38474d51c247c4e80544d5964
size 1150936

4
docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e9aca69f7a8aba308eed0932bb52ca703329c167786b855c33392c196775e3c7
size 116335
oid sha256:0541792d494f341f9b36f073472c0f6bc248297506ada281f7dfbb3feaf0a5af
size 120137

4
docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a72aac750b87ad7318f3caad198955aea63148e2bfa01db73fc7ccb9585304f8
size 1151306
oid sha256:1273f178881d3158052a8c494b40a5bb426a972130242bc0f84145c5bb2d98d6
size 1150777

238
docs/notebooks/language-quantize-bert-with-output.rst
View File

@ -50,15 +50,7 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -98,14 +90,10 @@ Imports
.. parsed-literal::
2024-04-18 00:02:10.199327: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-18 00:02:10.234239: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:20:31.324929: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:20:31.359787: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-18 00:02:10.832965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-07 00:20:31.955321: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@ -207,12 +195,12 @@ PyTorch model formats are supported:
.. parsed-literal::
WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.2.2+cpu. If you encounter issues, consider switching to torch==2.1.2
WARNING:nncf:NNCF provides best results with torch==2.2.*, while current torch version is 2.3.0+cpu. If you encounter issues, consider switching to torch==2.2.*
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
@ -326,10 +314,6 @@ The optimization process contains the following steps:
.. parsed-literal::
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
.. parsed-literal::
INFO:nncf:50 ignored nodes were found by name in the NNCFGraph
@ -492,16 +476,8 @@ Compare F1-score of FP32 and INT8 models
.. parsed-literal::
Checking the accuracy of the original model:
.. parsed-literal::
F1 score: 0.9019
Checking the accuracy of the quantized model:
.. parsed-literal::
F1 score: 0.8969
@ -556,17 +532,9 @@ Frames Per Second (FPS) for images.
.. parsed-literal::
PyTorch model on CPU: 0.075 seconds per sentence, SPS: 13.34
.. parsed-literal::
IR FP32 model in OpenVINO Runtime/AUTO: 0.020 seconds per sentence, SPS: 48.84
.. parsed-literal::
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 113.21
PyTorch model on CPU: 0.072 seconds per sentence, SPS: 13.84
IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 47.97
OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 110.23
Finally, measure the inference performance of OpenVINO ``FP32`` and
@ -586,7 +554,7 @@ in OpenVINO.
.. code:: ipython3
# Inference FP32 model (OpenVINO IR)
!benchmark_app -m $ir_model_xml -shape [1,128],[1,128],[1,128] -d device.value -api sync
!benchmark_app -m $ir_model_xml -shape [1,128],[1,128],[1,128] -d {device.value} -api sync
.. parsed-literal::
@ -594,32 +562,97 @@ in OpenVINO.
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device device.value
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ ERROR ] Exception from src/inference/src/cpp/core.cpp:216:
Exception from src/inference/src/dev/core_impl.cpp:556:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
supported_properties = benchmark.core.get_property(device, properties.supported_properties())
RuntimeError: Exception from src/inference/src/cpp/core.cpp:216:
Exception from src/inference/src/dev/core_impl.cpp:556:
Device with "device" name is not registered in the OpenVINO Runtime
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 19.23 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,?]
[ INFO ] attention_mask , 36 (node: attention_mask) : i64 / [...] / [1,?]
[ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,128], '36': [1,128], 'token_type_ids': [1,128]
[ INFO ] Reshape model took 5.65 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,128]
[ INFO ] attention_mask , 36 (node: attention_mask) : i64 / [...] / [1,128]
[ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,128]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 376.44 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] MULTI_DEVICE_PRIORITIES: CPU
[ INFO ] CPU:
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] ENABLE_HYPER_THREADING: False
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] PERF_COUNT: NO
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
[ WARNING ] No input files were given for input '36'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
[ INFO ] Fill input 'input_ids' with random values
[ INFO ] Fill input '36' with random values
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 22.61 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 6217 iterations
[ INFO ] Duration: 120004.55 ms
[ INFO ] Latency:
[ INFO ] Median: 19.20 ms
[ INFO ] Average: 19.21 ms
[ INFO ] Min: 18.57 ms
[ INFO ] Max: 23.34 ms
[ INFO ] Throughput: 51.81 FPS
.. code:: ipython3
# Inference INT8 model (OpenVINO IR)
! benchmark_app -m $compressed_model_xml -shape [1,128],[1,128],[1,128] -d device.value -api sync
! benchmark_app -m $compressed_model_xml -shape [1,128],[1,128],[1,128] -d {device.value} -api sync
.. parsed-literal::
@ -627,24 +660,89 @@ in OpenVINO.
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device device.value
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ ERROR ] Exception from src/inference/src/cpp/core.cpp:216:
Exception from src/inference/src/dev/core_impl.cpp:556:
Device with "device" name is not registered in the OpenVINO Runtime
Traceback (most recent call last):
File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
supported_properties = benchmark.core.get_property(device, properties.supported_properties())
RuntimeError: Exception from src/inference/src/cpp/core.cpp:216:
Exception from src/inference/src/dev/core_impl.cpp:556:
Device with "device" name is not registered in the OpenVINO Runtime
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 24.76 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,?]
[ INFO ] 36 , attention_mask (node: attention_mask) : i64 / [...] / [1,?]
[ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,128], '36': [1,128], 'token_type_ids': [1,128]
[ INFO ] Reshape model took 7.38 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,128]
[ INFO ] 36 , attention_mask (node: attention_mask) : i64 / [...] / [1,128]
[ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,128]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1183.71 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] MULTI_DEVICE_PRIORITIES: CPU
[ INFO ] CPU:
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] ENABLE_HYPER_THREADING: False
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] INFERENCE_NUM_THREADS: 12
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ] PERFORMANCE_HINT: LATENCY
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] PERF_COUNT: NO
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
[ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
[ WARNING ] No input files were given for input '36'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
[ INFO ] Fill input 'input_ids' with random values
[ INFO ] Fill input '36' with random values
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 15.91 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 11978 iterations
[ INFO ] Duration: 120006.11 ms
[ INFO ] Latency:
[ INFO ] Median: 10.29 ms
[ INFO ] Average: 9.93 ms
[ INFO ] Min: 8.15 ms
[ INFO ] Max: 11.91 ms
[ INFO ] Throughput: 99.81 FPS

869
docs/notebooks/latency-tricks-with-output.rst
View File

@ -1,869 +0,0 @@
Performance tricks in OpenVINO for latency mode
===============================================
The goal of this notebook is to provide a step-by-step tutorial for
improving performance for inferencing in a latency mode. Low latency is
especially desired in real-time applications when the results are needed
as soon as possible after the data appears. This notebook assumes
computer vision workflow and uses
`YOLOv5n <https://github.com/ultralytics/yolov5>`__ model. We will
simulate a camera application that provides frames one by one.
The performance tips applied in this notebook could be summarized in the
following figure. Some of the steps below can be applied to any device
at any stage, e.g., ``shared_memory``; some can be used only to specific
devices, e.g., ``INFERENCE_NUM_THREADS`` to CPU. As the number of
potential configurations is vast, we recommend looking at the steps
below and then apply a trial-and-error approach. You can incorporate
many hints simultaneously, like more inference threads + shared memory.
It should give even better performance, but we recommend testing it
anyway.
**NOTE**: We especially recommend trying
``OpenVINO IR model + CPU + shared memory in latency mode`` or
``OpenVINO IR model + CPU + shared memory + more inference threads``.
The quantization and pre-post-processing API are not included here as
they change the precision (quantization) or processing graph
(prepostprocessor). You can find examples of how to apply them to
optimize performance on OpenVINO IR files in
`optimize-preprocessing <../optimize-preprocessing>`__.
|image0|
**NOTE**: Many of the steps presented below will give you better
performance. However, some of them may **not change anything** or
even **worsen the performance** if they are strongly dependent on
either the hardware or the model. Please run this notebook on your
computer with your model to learn which of them makes sense in your
case.
All the following tricks were run with OpenVINO 2023.0. Future
versions of OpenVINO may include various optimizations that may
result in different performance.
A similar notebook focused on the throughput mode is available
`here <throughput-tricks.ipynb>`__.
Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
- `Data <#data>`__
- `Model <#model>`__
- `Hardware <#hardware>`__
- `Helper functions <#helper-functions>`__
- `Optimizations <#optimizations>`__
- `PyTorch model <#pytorch-model>`__
- `ONNX model <#onnx-model>`__
- `OpenVINO IR model <#openvino-ir-model>`__
- `OpenVINO IR model on GPU <#openvino-ir-model-on-gpu>`__
- `OpenVINO IR model + more inference
threads <#openvino-ir-model--more-inference-threads>`__
- `OpenVINO IR model in latency
mode <#openvino-ir-model-in-latency-mode>`__
- `OpenVINO IR model in latency mode + shared
memory <#openvino-ir-model-in-latency-mode--shared-memory>`__
- `Other tricks <#other-tricks>`__
- `Performance comparison <#performance-comparison>`__
- `Conclusions <#conclusions>`__
.. |image0| image:: https://user-images.githubusercontent.com/4547501/229120774-01f4f972-424d-4280-8395-220dd432985a.png
Prerequisites
-------------
.. code:: ipython3
import platform
%pip install -q "openvino>=2023.1.0" seaborn "ultralytics<=8.0.178" onnx opencv-python --extra-index-url https://download.pytorch.org/whl/cpu
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
%pip install -q "matplotlib>=3.4,<3.7"
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. code:: ipython3
import os
import time
from pathlib import Path
from typing import Any, List, Tuple
# Fetch `notebook_utils` module
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
import notebook_utils as utils
Data
----
We will use the same image of the dog sitting on a bicycle for all
experiments below. The image is resized and preprocessed to fulfill the
requirements of this particular object detection model.
.. code:: ipython3
import numpy as np
import cv2
IMAGE_WIDTH = 640
IMAGE_HEIGHT = 480
# load image
image = utils.load_image("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg")
image = cv2.resize(image, dsize=(IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_AREA)
# preprocess it for YOLOv5
input_image = image / 255.0
input_image = np.transpose(input_image, axes=(2, 0, 1))
input_image = np.expand_dims(input_image, axis=0)
# show the image
utils.show_array(image)
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg
.. parsed-literal::
<DisplayHandle display_id=301daec20bc62a0f381ba3798513e9eb>
Model
-----
We decided to go with
`YOLOv5n <https://github.com/ultralytics/yolov5>`__, one of the
state-of-the-art object detection models, easily available through the
PyTorch Hub and small enough to see the difference in performance.
.. code:: ipython3
import torch
from IPython.utils import io
# directory for all models
base_model_dir = Path("model")
model_name = "yolov5n"
model_path = base_model_dir / model_name
# load YOLOv5n from PyTorch Hub
pytorch_model = torch.hub.load("ultralytics/yolov5", "custom", path=model_path, device="cpu", skip_validation=True)
# don't print full model architecture
with io.capture_output():
pytorch_model.eval()
.. parsed-literal::
Using cache found in /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master
.. parsed-literal::
YOLOv5 🚀 2023-4-21 Python-3.8.10 torch-2.2.2+cpu CPU
.. parsed-literal::
requirements: /opt/home/k8sworker/.cache/torch/hub/requirements.txt not found, check failed.
.. parsed-literal::
Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5n.pt to model/yolov5n.pt...
.. parsed-literal::
0%| | 0.00/3.87M [00:00<?, ?B/s]
.. parsed-literal::
12%|█▏ | 480k/3.87M [00:00<00:00, 4.89MB/s]
.. parsed-literal::
81%|████████ | 3.14M/3.87M [00:00<00:00, 18.5MB/s]
.. parsed-literal::
100%|██████████| 3.87M/3.87M [00:00<00:00, 19.5MB/s]
.. parsed-literal::
Fusing layers...
.. parsed-literal::
YOLOv5n summary: 213 layers, 1867405 parameters, 0 gradients, 4.5 GFLOPs
.. parsed-literal::
Adding AutoShape...
Hardware
--------
The code below lists the available hardware we will use in the
benchmarking process.
**NOTE**: The hardware you have is probably completely different from
ours. It means you can see completely different results.
.. code:: ipython3
import openvino as ov
# initialize OpenVINO
core = ov.Core()
# print available devices
for device in core.available_devices:
device_name = core.get_property(device, "FULL_DEVICE_NAME")
print(f"{device}: {device_name}")
.. parsed-literal::
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
Helper functions
----------------
We’re defining a benchmark model function to use for all optimized
models below. It runs inference 1000 times, averages the latency time,
and prints two measures: seconds per image and frames per second (FPS).
.. code:: ipython3
INFER_NUMBER = 1000
def benchmark_model(model: Any, input_data: np.ndarray, benchmark_name: str, device_name: str = "CPU") -> float:
"""
Helper function for benchmarking the model. It measures the time and prints results.
"""
# measure the first inference separately - it may be slower as it contains also initialization
start = time.perf_counter()
model(input_data)
end = time.perf_counter()
first_infer_time = end - start
print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")
# benchmarking
start = time.perf_counter()
for _ in range(INFER_NUMBER):
model(input_data)
end = time.perf_counter()
# elapsed time
infer_time = end - start
# print second per image and FPS
mean_infer_time = infer_time / INFER_NUMBER
mean_fps = INFER_NUMBER / infer_time
print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")
return mean_infer_time
The following functions aim to post-process results and draw boxes on
the image.
.. code:: ipython3
# https://gist.github.com/AruniRC/7b3dadd004da04c80198557db5da4bda
classes = [
"person",
"bicycle",
"car",
"motorcycle",
"airplane",
"bus",
"train",
"truck",
"boat",
"traffic light",
"fire hydrant",
"stop sign",
"parking meter",
"bench",
"bird",
"cat",
"dog",
"horse",
"sheep",
"cow",
"elephant",
"bear",
"zebra",
"giraffe",
"backpack",
"umbrella",
"handbag",
"tie",
"suitcase",
"frisbee",
"skis",
"snowboard",
"sports ball",
"kite",
"baseball bat",
"baseball glove",
"skateboard",
"surfboard",
"tennis racket",
"bottle",
"wine glass",
"cup",
"fork",
"knife",
"spoon",
"bowl",
"banana",
"apple",
"sandwich",
"orange",
"broccoli",
"carrot",
"hot dog",
"pizza",
"donut",
"cake",
"chair",
"couch",
"potted plant",
"bed",
"dining table",
"toilet",
"tv",
"laptop",
"mouse",
"remote",
"keyboard",
"cell phone",
"microwave",
"oven",
"oaster",
"sink",
"refrigerator",
"book",
"clock",
"vase",
"scissors",
"teddy bear",
"hair drier",
"toothbrush",
]
# Colors for the classes above (Rainbow Color Map).
colors = cv2.applyColorMap(
src=np.arange(0, 255, 255 / len(classes), dtype=np.float32).astype(np.uint8),
colormap=cv2.COLORMAP_RAINBOW,
).squeeze()
def postprocess(detections: np.ndarray) -> List[Tuple]:
"""
Postprocess the raw results from the model.
"""
# candidates - probability > 0.25
detections = detections[detections[..., 4] > 0.25]
boxes = []
labels = []
scores = []
for obj in detections:
xmin, ymin, ww, hh = obj[:4]
score = obj[4]
label = np.argmax(obj[5:])
# Create a box with pixels coordinates from the box with normalized coordinates [0,1].
boxes.append(tuple(map(int, (xmin - ww // 2, ymin - hh // 2, ww, hh))))
labels.append(int(label))
scores.append(float(score))
# Apply non-maximum suppression to get rid of many overlapping entities.
# See https://paperswithcode.com/method/non-maximum-suppression
# This algorithm returns indices of objects to keep.
indices = cv2.dnn.NMSBoxes(bboxes=boxes, scores=scores, score_threshold=0.25, nms_threshold=0.5)
# If there are no boxes.
if len(indices) == 0:
return []
# Filter detected objects.
return [(labels[idx], scores[idx], boxes[idx]) for idx in indices.flatten()]
def draw_boxes(img: np.ndarray, boxes):
"""
Draw detected boxes on the image.
"""
for label, score, box in boxes:
# Choose color for the label.
color = tuple(map(int, colors[label]))
# Draw a box.
x2 = box[0] + box[2]
y2 = box[1] + box[3]
cv2.rectangle(img=img, pt1=box[:2], pt2=(x2, y2), color=color, thickness=2)
# Draw a label name inside the box.
cv2.putText(
img=img,
text=f"{classes[label]} {score:.2f}",
org=(box[0] + 10, box[1] + 20),
fontFace=cv2.FONT_HERSHEY_COMPLEX,
fontScale=img.shape[1] / 1200,
color=color,
thickness=1,
lineType=cv2.LINE_AA,
)
def show_result(results: np.ndarray):
"""
Postprocess the raw results, draw boxes and show the image.
"""
output_img = image.copy()
detections = postprocess(results)
draw_boxes(output_img, detections)
utils.show_array(output_img)
Optimizations
-------------
Below, we present the performance tricks for faster inference in the
latency mode. We release resources after every benchmarking to be sure
the same amount of resource is available for every experiment.
PyTorch model
~~~~~~~~~~~~~
First, we’re benchmarking the original PyTorch model without any
optimizations applied. We will treat it as our baseline.
.. code:: ipython3
import torch
with torch.no_grad():
result = pytorch_model(torch.as_tensor(input_image)).detach().numpy()[0]
show_result(result)
pytorch_infer_time = benchmark_model(
pytorch_model,
input_data=torch.as_tensor(input_image).float(),
benchmark_name="PyTorch model",
)
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg
.. parsed-literal::
PyTorch model on CPU. First inference time: 0.0269 seconds
.. parsed-literal::
PyTorch model on CPU: 0.0213 seconds per image (47.06 FPS)
ONNX model
~~~~~~~~~~
The first optimization is exporting the PyTorch model to ONNX and
running it in OpenVINO. It’s possible, thanks to the ONNX frontend. It
means we don’t necessarily have to convert the model to Intermediate
Representation (IR) to leverage the OpenVINO Runtime.
.. code:: ipython3
onnx_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}").with_suffix(".onnx")
# export PyTorch model to ONNX if it doesn't already exist
if not onnx_path.exists():
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(pytorch_model, dummy_input, onnx_path)
# load and compile in OpenVINO
onnx_model = core.read_model(onnx_path)
onnx_model = core.compile_model(onnx_model, device_name="CPU")
.. parsed-literal::
/opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:514: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
/opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py:64: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:
.. code:: ipython3
result = onnx_model(input_image)[onnx_model.output(0)][0]
show_result(result)
onnx_infer_time = benchmark_model(model=onnx_model, input_data=input_image, benchmark_name="ONNX model")
del onnx_model # release resources
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg
.. parsed-literal::
ONNX model on CPU. First inference time: 0.0173 seconds
.. parsed-literal::
ONNX model on CPU: 0.0136 seconds per image (73.79 FPS)
OpenVINO IR model
~~~~~~~~~~~~~~~~~
Let’s convert the ONNX model to OpenVINO Intermediate Representation
(IR) FP16 and run it. Reducing the precision is one of the well-known
methods for faster inference provided the hardware that supports lower
precision, such as FP16 or even INT8. If the hardware doesn’t support
lower precision, the model will be inferred in FP32 automatically. We
could also use quantization (INT8), but we should experience a little
accuracy drop. That’s why we skip that step in this notebook.
.. code:: ipython3
ov_model = ov.convert_model(onnx_path)
# save the model on disk
ov.save_model(ov_model, str(onnx_path.with_suffix(".xml")))
ov_cpu_model = core.compile_model(ov_model, device_name="CPU")
result = ov_cpu_model(input_image)[ov_cpu_model.output(0)][0]
show_result(result)
ov_cpu_infer_time = benchmark_model(model=ov_cpu_model, input_data=input_image, benchmark_name="OpenVINO model")
del ov_cpu_model # release resources
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg
.. parsed-literal::
OpenVINO model on CPU. First inference time: 0.0157 seconds
.. parsed-literal::
OpenVINO model on CPU: 0.0122 seconds per image (81.74 FPS)
OpenVINO IR model on GPU
~~~~~~~~~~~~~~~~~~~~~~~~
Usually, a GPU device is faster than a CPU, so let’s run the above model
on the GPU. Please note you need to have an Intel GPU and `install
drivers <https://github.com/openvinotoolkit/openvino_notebooks/wiki/Ubuntu#1-install-python-git-and-gpu-drivers-optional>`__
to be able to run this step. In addition, offloading to the GPU helps
reduce CPU load and memory consumption, allowing it to be left for
routine processes. If you cannot observe a faster inference on GPU, it
may be because the model is too light to benefit from massive parallel
execution.
.. code:: ipython3
ov_gpu_infer_time = 0.0
if "GPU" in core.available_devices:
ov_gpu_model = core.compile_model(ov_model, device_name="GPU")
result = ov_gpu_model(input_image)[ov_gpu_model.output(0)][0]
show_result(result)
ov_gpu_infer_time = benchmark_model(
model=ov_gpu_model,
input_data=input_image,
benchmark_name="OpenVINO model",
device_name="GPU",
)
del ov_gpu_model # release resources
OpenVINO IR model + more inference threads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There is a possibility to add a config for any device (CPU in this
case). We will increase the number of threads to an equal number of our
cores. There are `more
options <https://docs.openvino.ai/2024/api/c_cpp_api/group__ov__runtime__cpp__prop__api.html>`__
to be changed, so it’s worth playing with them to see what works best in
our case. In some cases, this optimization may worsen the performance.
If it is the case, don’t use it.
.. code:: ipython3
num_cores = os.cpu_count()
ov_cpu_config_model = core.compile_model(ov_model, device_name="CPU", config={"INFERENCE_NUM_THREADS": num_cores})
result = ov_cpu_config_model(input_image)[ov_cpu_config_model.output(0)][0]
show_result(result)
ov_cpu_config_infer_time = benchmark_model(
model=ov_cpu_config_model,
input_data=input_image,
benchmark_name="OpenVINO model + more threads",
)
del ov_cpu_config_model # release resources
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg
.. parsed-literal::
OpenVINO model + more threads on CPU. First inference time: 0.0157 seconds
.. parsed-literal::
OpenVINO model + more threads on CPU: 0.0123 seconds per image (81.20 FPS)
OpenVINO IR model in latency mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO offers a virtual device called
`AUTO <https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.html>`__,
which can select the best device for us based on a performance hint.
There are three different hints: ``LATENCY``, ``THROUGHPUT``, and
``CUMULATIVE_THROUGHPUT``. As this notebook is focused on the latency
mode, we will use ``LATENCY``. The above hints can be used with other
devices as well.
.. code:: ipython3
ov_auto_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "LATENCY"})
result = ov_auto_model(input_image)[ov_auto_model.output(0)][0]
show_result(result)
ov_auto_infer_time = benchmark_model(
model=ov_auto_model,
input_data=input_image,
benchmark_name="OpenVINO model",
device_name="AUTO",
)
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg
.. parsed-literal::
OpenVINO model on AUTO. First inference time: 0.0157 seconds
.. parsed-literal::
OpenVINO model on AUTO: 0.0124 seconds per image (80.65 FPS)
OpenVINO IR model in latency mode + shared memory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenVINO is a C++ toolkit with Python wrappers (API). The default
behavior in the Python API is copying the input to the additional buffer
and then running processing in C++, which prevents many
multiprocessing-related issues. However, it also increases time cost. We
can create a tensor with enabled shared memory (keeping in mind we
cannot overwrite our input), save time for copying and improve the
performance!
.. code:: ipython3
# it must be assigned to a variable, not to be garbage collected
c_input_image = np.ascontiguousarray(input_image, dtype=np.float32)
input_tensor = ov.Tensor(c_input_image, shared_memory=True)
result = ov_auto_model(input_tensor)[ov_auto_model.output(0)][0]
show_result(result)
ov_auto_shared_infer_time = benchmark_model(
model=ov_auto_model,
input_data=input_tensor,
benchmark_name="OpenVINO model + shared memory",
device_name="AUTO",
)
del ov_auto_model # release resources
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg
.. parsed-literal::
OpenVINO model + shared memory on AUTO. First inference time: 0.0113 seconds
.. parsed-literal::
OpenVINO model + shared memory on AUTO: 0.0054 seconds per image (186.74 FPS)
Other tricks
~~~~~~~~~~~~
There are other tricks for performance improvement, such as quantization
and pre-post-processing or dedicated to throughput mode. To get even
more from your model, please visit
`optimize-preprocessing <../optimize-preprocessing>`__, and
`throughput-tricks <throughput-tricks.ipynb>`__.
Performance comparison
----------------------
The following graphical comparison is valid for the selected model and
hardware simultaneously. If you cannot see any improvement between some
steps, just skip them.
.. code:: ipython3
%matplotlib inline
.. code:: ipython3
from matplotlib import pyplot as plt
labels = [
"PyTorch model",
"ONNX model",
"OpenVINO IR model",
"OpenVINO IR model on GPU",
"OpenVINO IR model + more inference threads",
"OpenVINO IR model in latency mode",
"OpenVINO IR model in latency mode + shared memory",
]
# make them milliseconds
times = list(
map(
lambda x: 1000 * x,
[
pytorch_infer_time,
onnx_infer_time,
ov_cpu_infer_time,
ov_gpu_infer_time,
ov_cpu_config_infer_time,
ov_auto_infer_time,
ov_auto_shared_infer_time,
],
)
)
bar_colors = colors[::10] / 255.0
fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(labels, times, color=bar_colors)
ax.set_ylabel("Inference time [ms]")
ax.set_title("Performance difference")
plt.xticks(rotation="vertical")
plt.show()
.. image:: latency-tricks-with-output_files/latency-tricks-with-output_31_0.png
Conclusions
-----------
We already showed the steps needed to improve the performance of an
object detection model. Even if you experience much better performance
after running this notebook, please note this may not be valid for every
hardware or every model. For the most accurate results, please use
``benchmark_app`` `command-line
tool <https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-tool.html>`__.
Note that ``benchmark_app`` cannot measure the impact of some tricks
above, e.g., shared memory.

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
size 162715

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_31_0.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e50d8a22c26feebfe2a2d322d21b370b0610cbf00740d0a5808a2644d62f7028
size 57013

3
docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:41c502fdff24ada81c63ccfca7d9153ea368b1eb3330caa03afa3281c35e4484
size 155828

8
docs/notebooks/latent-consistency-models-image-generation-with-output.rst
View File

@ -553,14 +553,6 @@ decoded by the decoder part of the variational auto encoder.
return_tensors="pt",
)
text_input_ids = text_inputs.input_ids
untruncated_ids = self.tokenizer(prompt, padding="longest", return_tensors="pt").input_ids
if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(text_input_ids, untruncated_ids):
removed_text = self.tokenizer.batch_decode(untruncated_ids[:, self.tokenizer.model_max_length - 1 : -1])
logger.warning(
"The following part of your input was truncated because CLIP can only handle sequences up to"
f" {self.tokenizer.model_max_length} tokens: {removed_text}"
)
prompt_embeds = self.text_encoder(text_input_ids, share_inputs=True, share_outputs=True)
prompt_embeds = torch.from_numpy(prompt_embeds[0])

109
docs/notebooks/latent-consistency-models-optimum-demo-with-output.rst
View File

@ -56,15 +56,7 @@ Install required packages
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@ -129,14 +121,10 @@ https://huggingface.co/docs/diffusers/en/api/pipelines/latent_consistency_models
.. parsed-literal::
2024-04-18 00:04:08.916564: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-18 00:04:08.953118: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-07 00:26:31.942146: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-07 00:26:31.978132: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-18 00:04:09.550802: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-07 00:26:32.469596: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@ -149,7 +137,7 @@ https://huggingface.co/docs/diffusers/en/api/pipelines/latent_consistency_models
prompt = "A cute squirrel in the forest, portrait, 8k"
image = pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0).images[0]
image = pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0, height=512, width=512).images[0]
image.save("image_standard_pipeline.png")
image
@ -217,10 +205,15 @@ and there is no need to do it manually
.. code:: ipython3
from optimum.intel.openvino import OVLatentConsistencyModelPipeline
from pathlib import Path
ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", export=True, compile=False)
ov_pipeline.reshape(batch_size=1, height=768, width=768, num_images_per_prompt=1)
ov_pipeline.save_pretrained("./openvino_ir")
if not Path("./openvino_ir").exists():
ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", height=512, width=512, export=True, compile=False)
ov_pipeline.save_pretrained("./openvino_ir")
else:
ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("./openvino_ir", export=False, compile=False)
ov_pipeline.reshape(batch_size=1, height=512, width=512, num_images_per_prompt=1)
.. parsed-literal::
@ -231,11 +224,7 @@ and there is no need to do it manually
.. parsed-literal::
Framework not specified. Using pt to export the model.
.. parsed-literal::
Keyword arguments {'subfolder': '', 'trust_remote_code': False} are not expected by StableDiffusionPipeline and will be ignored.
Keyword arguments {'subfolder': '', 'token': None, 'trust_remote_code': False} are not expected by StableDiffusionPipeline and will be ignored.
@ -246,7 +235,7 @@ and there is no need to do it manually
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
Using framework PyTorch: 2.3.0+cpu
.. parsed-literal::
@ -257,22 +246,58 @@ and there is no need to do it manually
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
Using framework PyTorch: 2.3.0+cpu
Using framework PyTorch: 2.3.0+cpu
Using framework PyTorch: 2.3.0+cpu
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
OVLatentConsistencyModelPipeline {
"_class_name": "OVLatentConsistencyModelPipeline",
"_diffusers_version": "0.24.0",
"feature_extractor": [
"transformers",
"CLIPImageProcessor"
],
"requires_safety_checker": true,
"safety_checker": [
"stable_diffusion",
"StableDiffusionSafetyChecker"
],
"scheduler": [
"diffusers",
"LCMScheduler"
],
"text_encoder": [
"optimum",
"OVModelTextEncoder"
],
"text_encoder_2": [
null,
null
],
"tokenizer": [
"transformers",
"CLIPTokenizer"
],
"unet": [
"optimum",
"OVModelUnet"
],
"vae_decoder": [
"optimum",
"OVModelVaeDecoder"
],
"vae_encoder": [
"optimum",
"OVModelVaeEncoder"
]
}
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
.. parsed-literal::
Using framework PyTorch: 2.2.2+cpu
.. code:: ipython3
@ -283,20 +308,8 @@ and there is no need to do it manually
.. parsed-literal::
Compiling the vae_decoder to CPU ...
.. parsed-literal::
Compiling the unet to CPU ...
.. parsed-literal::
Compiling the vae_encoder to CPU ...
.. parsed-literal::
Compiling the text_encoder to CPU ...
@ -304,7 +317,7 @@ and there is no need to do it manually
prompt = "A cute squirrel in the forest, portrait, 8k"
image_ov = ov_pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0).images[0]
image_ov = ov_pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0, height=512, width=512).images[0]
image_ov.save("image_opt.png")
image_ov

4
docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:978131b23cda8ee5f97efcd3c11e8853158abb75b93be87c596b70987c0abab4
size 77604
oid sha256:f55f400d3c6132e0208b817685c99756778f625b48a68f0a061909eb21b17b73
size 30125

4
docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:156d18fc303a7c9a92e356d1da39456c34ee8719291cc88cb225e6081c6c8a37
size 1004233
oid sha256:b73d5a12608ef2be6bed0c105a19e7375b187853f502d6659c3fd84e0d968759
size 416282

4
docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.jpg
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b22f2298a9e2ddfd16462395c9d44657c7e8fd719eac2376f2dde5799e18a311
size 74573
oid sha256:66b5aca06c991e57d8cf17b73f6260878086185c9a7f5901d054fe53637adf3a
size 36310

4
docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.png
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:36c4842ee6b9a0232ddc779aa24e05bcca0dfd9bfa018a4e77c9c990e59b4383
size 1004628
oid sha256:bfb27d57b247ea9ea90b3a6d194f3ad1d874c657ad6ec204b0fffc0623c09442
size 454841

132
docs/notebooks/legacy-mo-convert-to-openvino-with-output.rst
View File

@ -119,7 +119,7 @@ documentation.
conversion into IR. The legacy Frontend is Python
based and is available for TensorFlow*, ONNX*, MXNet*,
Caffe*, and Kaldi* models.
--input_model INPUT_MODEL, -w INPUT_MODEL, -m INPUT_MODEL
--input_model INPUT_MODEL, -m INPUT_MODEL, -w INPUT_MODEL
Tensorflow*: a file with a pre-trained model (binary
or text .pb file after freezing). Caffe*: a model
proto file with model weights.
@ -735,19 +735,13 @@ NLP model from Hugging Face and export it in ONNX format:
.. parsed-literal::
2024-04-17 23:28:06.943236: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-17 23:28:06.978999: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-05-06 23:47:34.384632: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-06 23:47:34.420652: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
.. parsed-literal::
2024-04-17 23:28:07.628516: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
2024-05-06 23:47:35.064940: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)
@ -1009,13 +1003,9 @@ To convert a model to OpenVINO IR, use the following command:
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@ -1109,13 +1099,9 @@ guide <https://docs.openvino.ai/2024/openvino-workflow/model-preparation/setting
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. parsed-literal::
@ -1132,13 +1118,9 @@ guide <https://docs.openvino.ai/2024/openvino-workflow/model-preparation/setting
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@ -1185,13 +1167,9 @@ sequence length dimension for inputs:
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@ -1234,13 +1212,9 @@ dimension:
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@ -1306,13 +1280,9 @@ guide <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. parsed-literal::
@ -1329,13 +1299,9 @@ guide <https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_prepare_model_convert
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@ -1409,13 +1375,9 @@ Resnet50 model that was exported to the ONNX format:
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. code:: ipython3
@ -1459,13 +1421,9 @@ presented by input data. Use either ``layout`` or ``source_layout`` with
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. parsed-literal::
@ -1482,13 +1440,9 @@ presented by input data. Use either ``layout`` or ``source_layout`` with
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. code:: ipython3
@ -1536,13 +1490,9 @@ that the preprocessing takes negligible time for inference.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. parsed-literal::
@ -1559,13 +1509,9 @@ that the preprocessing takes negligible time for inference.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. code:: ipython3
@ -1610,13 +1556,9 @@ the color channels before inference.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. code:: ipython3
@ -1659,13 +1601,9 @@ models, this decrease is negligible.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
.. parsed-literal::
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/resnet.bin
.. code:: ipython3

4
docs/notebooks/llava-next-multimodal-chatbot-with-output.rst
View File

@ -645,7 +645,7 @@ inference faster. The optimization process contains the following steps:
.. code:: ipython3
IMAGE_ENCODER_PATH_INT8 = IMAGE_ENCODER_PATH.parent / IMAGE_ENCODER_PATH.name.replace(".xml", "-int4.xml")
IMAGE_ENCODER_PATH_INT8 = IMAGE_ENCODER_PATH.parent / IMAGE_ENCODER_PATH.name.replace(".xml", "-int8.xml")
import requests
@ -1224,7 +1224,7 @@ Select device
use_int8_image_encoder = widgets.Checkbox(
value=IMAGE_ENCODER_PATH_INT8.exists(),
description="INT4 language model",
description="INT8 image encoder",
disabled=not IMAGE_ENCODER_PATH_INT8.exists(),
)

72
docs/notebooks/llm-agent-langchain-with-output.rst
View File

@ -220,7 +220,7 @@ folder.
model_path = "neural-chat-7b-v3-1-ov-int4"
if not Path(model_path).exists():
!optimum-cli export openvino --model ${model_id} --weight-format int4 ${model_path}
!optimum-cli export openvino --model {model_id} --weight-format int4 {model_path}
Select inference device for LLM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -247,7 +247,7 @@ Select inference device for LLM
.. parsed-literal::
Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')
Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')
@ -257,9 +257,54 @@ class in LangChain. To deploy a model with OpenVINO, you can specify the
inference framework. For `more
information <https://python.langchain.com/docs/integrations/llms/openvino/>`__.
You can get additional inference speed improvement with `Dynamic
Quantization of activations and KV-cache
quantization <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-hf.html#enabling-openvino-runtime-optimizations>`__.
.. code:: ipython3
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
ov_llm = HuggingFacePipeline.from_model_id(
model_id=model_path,
task="text-generation",
backend="openvino",
model_kwargs={"device": device.value, "ov_config": ov_config},
pipeline_kwargs={"max_new_tokens": 1024},
)
.. parsed-literal::
2024-05-01 12:57:42.013703: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-01 12:57:42.015389: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 12:57:42.049792: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-05-01 12:57:42.050591: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-01 12:57:42.819557: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
.. parsed-literal::
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
Python 3.8.18 (you have 3.8.10)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Compiling the model to CPU ...
You can get additional inference speed improvement with [Dynamic
Quantization of activations and KV-cache quantization] on
CPU(https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-hf.html#enabling-openvino-runtime-optimizations).
These options can be enabled with ``ov_config`` as follows:
.. code:: ipython3
@ -272,18 +317,6 @@ These options can be enabled with ``ov_config`` as follows:
"CACHE_DIR": "",
}
.. code:: ipython3
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
ov_llm = HuggingFacePipeline.from_model_id(
model_id=model_path,
task="text-generation",
backend="openvino",
model_kwargs={"device": device.value, "ov_config": ov_config},
pipeline_kwargs={"max_new_tokens": 1024},
)
Create agent
------------
@ -321,11 +354,6 @@ prompt template.
agent_executor.invoke({"input": "Take 3 to the fifth power and multiply that by the sum of twelve and three"})
.. parsed-literal::
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
.. parsed-literal::

349
docs/notebooks/llm-chatbot-with-output.rst
View File

@ -78,18 +78,18 @@ Install required dependencies
.. code:: ipython3
import shutil
import os
from pathlib import Path
import requests
# fetch model configuration
config_shared_path = Path("../../utils/llm_config.py")
config_dst_path = Path("llm_config.py")
if not config_dst_path.exists():
if config_shared_path.exists():
shutil.copy(config_shared_path, config_dst_path)
os.symlink(config_shared_path, config_dst_path)
else:
r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/llm_config.py")
with open("llm_config.py", "w") as f:
@ -146,7 +146,7 @@ The available options are:
.. code:: python
## login to huggingfacehub to get access to pretrained model
## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@ -156,6 +156,14 @@ The available options are:
except OSError:
notebook_login()
- **phi3-mini-instruct<|end|>** - The Phi-3-Mini is a 3.8B parameters,
lightweight, state-of-the-art open model trained with the Phi-3
datasets that includes both synthetic data and the filtered publicly
available websites data with a focus on high-quality and reasoning
dense properties. More details about model can be found in `model
card <https://huggingface.co/microsoft/Phi-3-mini-4k-instruct>`__,
`Microsoft blog <https://aka.ms/phi3blog-april>`__ and `technical
report <https://aka.ms/phi3-tech-report>`__.
- **red-pajama-3b-chat** - A 2.8B parameter pre-trained language model
based on GPT-NEOX architecture. It was developed by Together Computer
and leaders from the open-source AI community. The model is
@ -184,7 +192,7 @@ The available options are:
.. code:: python
## login to huggingfacehub to get access to pretrained model
## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@ -217,7 +225,41 @@ The available options are:
.. code:: python
## login to huggingfacehub to get access to pretrained model
## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
try:
whoami()
print('Authorization token already provided')
except OSError:
notebook_login()
- **llama-3-8b-instruct** - Llama 3 is an auto-regressive language
model that uses an optimized transformer architecture. The tuned
versions use supervised fine-tuning (SFT) and reinforcement learning
with human feedback (RLHF) to align with human preferences for
helpfulness and safety. The Llama 3 instruction tuned models are
optimized for dialogue use cases and outperform many of the available
open source chat models on common industry benchmarks. More details
about model can be found in `Meta blog
post <https://ai.meta.com/blog/meta-llama-3/>`__, `model
website <https://llama.meta.com/llama3>`__ and `model
card <https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>`__.
>\ **Note**: run model with demo, you will need to accept license
agreement. >You must be a registered user in Hugging Face Hub.
Please visit `HuggingFace model
card <https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>`__,
carefully read terms of usage and click accept button. You will need
to use an access token for the code below to run. For more
information on access tokens, refer to `this section of the
documentation <https://huggingface.co/docs/hub/security-tokens>`__.
>You can login on Hugging Face Hub in notebook environment, using
following code:
.. code:: python
## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@ -336,14 +378,14 @@ The available options are:
.. code:: ipython3
model_languages = list(SUPPORTED_LLM_MODELS)
model_language = widgets.Dropdown(
options=model_languages,
value=model_languages[0],
description="Model Language:",
disabled=False,
)
model_language
@ -358,14 +400,14 @@ The available options are:
.. code:: ipython3
model_ids = list(SUPPORTED_LLM_MODELS[model_language.value])
model_id = widgets.Dropdown(
options=model_ids,
value=model_ids[0],
value=model_ids[2],
description="Model:",
disabled=False,
)
model_id
@ -373,7 +415,7 @@ The available options are:
.. parsed-literal::
Dropdown(description='Model:', options=('tiny-llama-1b-chat', 'gemma-2b-it', 'red-pajama-3b-chat', 'gemma-7b-i…
Dropdown(description='Model:', index=2, options=('tiny-llama-1b-chat', 'gemma-2b-it', 'phi-3-mini-instruct', '…
@ -385,7 +427,7 @@ The available options are:
.. parsed-literal::
Selected model tiny-llama-1b-chat
Selected model phi-3-mini-instruct
Convert model using Optimum-CLI tool
@ -418,8 +460,7 @@ that exported model should solve. For LLMs it will be
``text-generation-with-past``. If model initialization requires to use
remote code, ``--trust-remote-code`` flag additionally should be passed.
Compress model weights
----------------------
<|end|>## Compress model weights
The `Weights
Compression <https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html>`__
@ -462,7 +503,7 @@ sacrifice of the model size and inference latency.
.. code:: ipython3
from IPython.display import Markdown, display
prepare_int4_model = widgets.Checkbox(
value=True,
description="Prepare INT4 model",
@ -478,7 +519,7 @@ sacrifice of the model size and inference latency.
description="Prepare FP16 model",
disabled=False,
)
display(prepare_int4_model)
display(prepare_int8_model)
display(prepare_fp16_model)
@ -507,14 +548,14 @@ We can now save floating point and compressed model variants
.. code:: ipython3
from pathlib import Path
pt_model_id = model_configuration["model_id"]
pt_model_name = model_id.value.split("-")[0]
fp16_model_dir = Path(model_id.value) / "FP16"
int8_model_dir = Path(model_id.value) / "INT8_compressed_weights"
int4_model_dir = Path(model_id.value) / "INT4_compressed_weights"
def convert_to_fp16():
if (fp16_model_dir / "openvino_model.xml").exists():
return
@ -526,8 +567,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
def convert_to_int8():
if (int8_model_dir / "openvino_model.xml").exists():
return
@ -540,8 +581,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
def convert_to_int4():
compression_configs = {
"zephyr-7b-beta": {
@ -579,6 +620,11 @@ We can now save floating point and compressed model variants
"group_size": 128,
"ratio": 0.8,
},
"llama-3-8b-instruct": {
"sym": True,
"group_size": 128,
"ratio": 0.8,
},
"gemma-7b-it": {
"sym": True,
"group_size": 128,
@ -601,7 +647,7 @@ We can now save floating point and compressed model variants
"ratio": 0.8,
},
}
model_compression_params = compression_configs.get(model_id.value, compression_configs["default"])
if (int4_model_dir / "openvino_model.xml").exists():
return
@ -617,8 +663,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
if prepare_fp16_model.value:
convert_to_fp16()
if prepare_int8_model.value:
@ -626,58 +672,6 @@ We can now save floating point and compressed model variants
if prepare_int4_model.value:
convert_to_int4()
**Export command:**
``optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --task text-generation-with-past --weight-format int4 --group-size 128 --ratio 0.8 tiny-llama-1b-chat/INT4_compressed_weights``
.. parsed-literal::
2024-04-11 11:48:29.180963: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-11 11:48:29.182830: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-11 11:48:29.219152: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-11 11:48:29.219549: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-11 11:48:29.930190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
Python 3.8.18 (you have 3.8.10)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Framework not specified. Using pt to export the model.
Using the export variant default. Available variants are:
- default: The default ONNX variant.
Using framework PyTorch: 2.1.2+cpu
Overriding 1 configuration item(s)
- use_cache -> True
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
The cos_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
The sin_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/optimum/exporters/openvino/model_patcher.py:311: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if sequence_length != 1:
Mixed-Precision assignment ━━━━━━━━━━━━━━━━━━━━ 100% 154/154 • 0:00:11 • 0:00:00;0;104;181m0:00:01181m0:00:01
INFO:nncf:Statistics of the bitwidth distribution:
┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │
┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ 8 │ 30% (42 / 156) │ 20% (40 / 154) │
├────────────────┼─────────────────────────────┼────────────────────────────────────────┤
│ 4 │ 70% (114 / 156) │ 80% (114 / 154) │
┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
Applying Weight Compression ━━━━━━━━━━━━━━━━━━━ 100% 156/156 • 0:00:26 • 0:00:00;0;104;181m0:00:01181m0:00:02
Let’s compare model size for different compression types
.. code:: ipython3
@ -685,7 +679,7 @@ Let’s compare model size for different compression types
fp16_weights = fp16_model_dir / "openvino_model.bin"
int8_weights = int8_model_dir / "openvino_model.bin"
int4_weights = int4_model_dir / "openvino_model.bin"
if fp16_weights.exists():
print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
@ -697,7 +691,7 @@ Let’s compare model size for different compression types
.. parsed-literal::
Size of model with INT4 compressed weights is 696.19 MB
Size of model with INT4 compressed weights is 2339.74 MB
Select device for inference and model variant
@ -711,16 +705,16 @@ Select device for inference and model variant
.. code:: ipython3
import openvino as ov
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="CPU",
description="Device:",
disabled=False,
)
device
@ -744,14 +738,14 @@ variant of model weights and inference device
available_models.append("INT8")
if fp16_model_dir.exists():
available_models.append("FP16")
model_to_run = widgets.Dropdown(
options=available_models,
value=available_models[0],
description="Model to run:",
disabled=False,
)
model_to_run
@ -803,7 +797,7 @@ guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__
from transformers import AutoConfig, AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
if model_to_run.value == "INT4":
model_dir = int4_model_dir
elif model_to_run.value == "INT8":
@ -811,17 +805,17 @@ guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__
else:
model_dir = fp16_model_dir
print(f"Loading model from {model_dir}")
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
# On a GPU device a model is executed in FP16 precision. For red-pajama-3b-chat model there known accuracy
# issues caused by this, which we avoid by setting precision hint to "f32".
if model_id.value == "red-pajama-3b-chat" and "GPU" in core.available_devices and device.value in ["GPU", "AUTO"]:
ov_config["INFERENCE_PRECISION_HINT"] = "f32"
model_name = model_configuration["model_id"]
tok = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
ov_model = OVModelForCausalLM.from_pretrained(
model_dir,
device=device.value,
@ -830,6 +824,52 @@ guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__
trust_remote_code=True,
)
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2024-04-23 22:13:04.208987: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-23 22:13:04.210866: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-23 22:13:04.245998: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-23 22:13:04.246894: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-23 22:13:04.941663: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
.. parsed-literal::
/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
.. parsed-literal::
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
Python 3.8.18 (you have 3.8.10)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
.. parsed-literal::
Loading model from phi-3-mini-instruct/INT4_compressed_weights
.. parsed-literal::
The argument `trust_remote_code` is to be used along with export=True. It will be ignored.
Compiling the model to CPU ...
.. code:: ipython3
tokenizer_kwargs = model_configuration.get("tokenizer_kwargs", {})
@ -839,11 +879,6 @@ guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__
print(tok.batch_decode(answer, skip_special_tokens=True)[0])
.. parsed-literal::
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
.. parsed-literal::
2 + 2 = 4
@ -889,14 +924,14 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
::
playing: 0.5
sleeping: 0.25
eating: 0.15
driving: 0.05
flying: 0.05
playing: 0.5
sleeping: 0.25
eating: 0.15
driving: 0.05
flying: 0.05
- **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as "playing."
- **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as "playing," "sleeping," or "eating."
- **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as "playing."
- **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as "playing," "sleeping," or "eating."
- **High temperature** (e.g., 2.0): The AI model becomes more adventurous, increasing the chances of selecting less likely tokens, such as "driving" and "flying."
- ``Top-p``, also known as nucleus sampling, is a parameter used to
@ -943,15 +978,15 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
StoppingCriteriaList,
TextIteratorStreamer,
)
model_name = model_configuration["model_id"]
start_message = model_configuration["start_message"]
history_template = model_configuration.get("history_template")
current_message_template = model_configuration.get("current_message_template")
stop_tokens = model_configuration.get("stop_tokens")
tokenizer_kwargs = model_configuration.get("tokenizer_kwargs", {})
chinese_examples = [
["你好!"],
["你是谁?"],
@ -961,7 +996,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["给我讲一个年轻人奋斗创业最终取得成功的故事。"],
["给这个故事起一个标题。"],
]
english_examples = [
["Hello there! How are you doing?"],
["What is OpenVINO?"],
@ -971,7 +1006,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["What are some common mistakes to avoid when writing code?"],
["Write a 100-word blog post on “Benefits of Artificial Intelligence and OpenVINO“"],
]
japanese_examples = [
["こんにちは！調子はどうですか?"],
["OpenVINOとは何ですか?"],
@ -981,48 +1016,48 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["コードを書くときに避けるべきよくある間違いは何ですか?"],
["人工知能と「OpenVINOの利点」について100語程度のブログ記事を書いてください。"],
]
examples = chinese_examples if (model_language.value == "Chinese") else japanese_examples if (model_language.value == "Japanese") else english_examples
max_new_tokens = 256
class StopOnTokens(StoppingCriteria):
def __init__(self, token_ids):
self.token_ids = token_ids
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
for stop_id in self.token_ids:
if input_ids[0][-1] == stop_id:
return True
return False
if stop_tokens is not None:
if isinstance(stop_tokens[0], str):
stop_tokens = tok.convert_tokens_to_ids(stop_tokens)
stop_tokens = [StopOnTokens(stop_tokens)]
def default_partial_text_processor(partial_text: str, new_text: str):
"""
helper for updating partially generated answer, used by default
Params:
partial_text: text buffer for storing previosly generated text
new_text: text update for the current step
Returns:
updated text string
"""
partial_text += new_text
return partial_text
text_processor = model_configuration.get("partial_text_processor", default_partial_text_processor)
def convert_history_to_token(history: List[Tuple[str, str]]):
"""
function for conversion history stored as list pairs of user and assistant messages to tokens according to model expected conversation template
@ -1056,7 +1091,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
messages.append({"role": "user", "content": user_msg})
if model_msg:
messages.append({"role": "assistant", "content": model_msg})
input_token = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt")
else:
text = start_message + "".join(
@ -1077,12 +1112,12 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
)
input_token = tok(text, return_tensors="pt", **tokenizer_kwargs).input_ids
return input_token
def user(message, history):
"""
callback function for updating user messages in interface on submit button click
Params:
message: current message
history: conversation history
@ -1091,12 +1126,12 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
"""
# Append the user's message to the conversation history
return "", history + [[message, ""]]
def bot(history, temperature, top_p, top_k, repetition_penalty, conversation_id):
"""
callback function for running chatbot on submit button click
Params:
history: conversation history
temperature: parameter for control the level of creativity in AI-generated text.
@ -1105,9 +1140,9 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
top_k: parameter for control the range of tokens considered by the AI model based on their cumulative probability, selecting number of tokens with highest probability.
repetition_penalty: parameter for penalizing tokens based on how frequently they occur in the text.
conversation_id: unique conversation identifier.
"""
# Construct the input message string for the model by concatenating the current system message and conversation history
# Tokenize the messages string
input_ids = convert_history_to_token(history)
@ -1127,9 +1162,9 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
)
if stop_tokens is not None:
generate_kwargs["stopping_criteria"] = StoppingCriteriaList(stop_tokens)
stream_complete = Event()
def generate_and_signal_complete():
"""
genration function for single thread
@ -1137,29 +1172,29 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
global start_time
ov_model.generate(**generate_kwargs)
stream_complete.set()
t1 = Thread(target=generate_and_signal_complete)
t1.start()
# Initialize an empty string to store the generated text
partial_text = ""
for new_text in streamer:
partial_text = text_processor(partial_text, new_text)
history[-1][1] = partial_text
yield history
def request_cancel():
ov_model.request.cancel()
def get_uuid():
"""
universal unique identifier for thread
"""
return str(uuid4())
with gr.Blocks(
theme=gr.themes.Soft(),
css=".disclaimer {font-variant-caps: all-small-caps;}",
@ -1231,7 +1266,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
info="Penalize repetition — 1.0 to disable.",
)
gr.Examples(examples, inputs=msg, label="Click on any example and press the 'Submit' button")
submit_event = msg.submit(
fn=user,
inputs=[msg, chatbot],
@ -1276,7 +1311,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
queue=False,
)
clear.click(lambda: None, None, chatbot, queue=False)
# if you are launching remotely, specify server_name and server_port
# demo.launch(server_name='your server name', server_port='server port in int')
# if you have any issue to launch on your platform, you can pass share=True to launch method:
@ -1284,6 +1319,26 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
# it creates a publicly shareable link for the interface. Read more in the docs: https://gradio.app/docs/
demo.launch()
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
.. code:: ipython3
# please uncomment and run this cell for stopping gradio interface

133
docs/notebooks/llm-question-answering-with-output.rst
View File

@ -128,9 +128,54 @@ The available options are:
card <https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2>`__,
`paper <https://arxiv.org/abs/2310.06825>`__ and `release blog
post <https://mistral.ai/news/announcing-mistral-7b/>`__.
- **llama-3-8b-instruct** - Llama 3 is an auto-regressive language
model that uses an optimized transformer architecture. The tuned
versions use supervised fine-tuning (SFT) and reinforcement learning
with human feedback (RLHF) to align with human preferences for
helpfulness and safety. The Llama 3 instruction tuned models are
optimized for dialogue use cases and outperform many of the available
open source chat models on common industry benchmarks. More details
about model can be found in `Meta blog
post <https://ai.meta.com/blog/meta-llama-3/>`__, `model
website <https://llama.meta.com/llama3>`__ and `model
card <https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>`__.
>\ **Note**: run model with demo, you will need to accept license
agreement. >You must be a registered user in Hugging Face Hub.
Please visit `HuggingFace model
card <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`__,
carefully read terms of usage and click accept button. You will need
to use an access token for the code below to run. For more
information on access tokens, refer to `this section of the
documentation <https://huggingface.co/docs/hub/security-tokens>`__.
>You can login on Hugging Face Hub in notebook environment, using
following code:
.. code:: python
## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
try:
whoami()
print('Authorization token already provided')
except OSError:
notebook_login()
.. code:: ipython3
from pathlib import Path
import requests
# Fetch `notebook_utils` module
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
if not Path("./config.py").exists():
download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-question-answering/config.py")
from config import SUPPORTED_LLM_MODELS
import ipywidgets as widgets
@ -164,7 +209,7 @@ The available options are:
.. parsed-literal::
Selected model phi-2
Selected model llama-3-8b-instruct
Instantiate Model using Optimum Intel
@ -307,10 +352,8 @@ compression.
import openvino as ov
import nncf
from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig
from optimum.utils import NormalizedTextConfig, NormalizedConfigManager
import gc
NormalizedConfigManager._conf["phi"] = NormalizedTextConfig
nncf.set_log_level(logging.ERROR)
@ -354,6 +397,7 @@ compression.
"ratio": 0.5,
},
"dolly-v2-3b": {"sym": False, "group_size": 32, "ratio": 0.5},
"llama-3-8b-instruct": {"sym": True, "group_size": 128, "ratio": 1.0},
"default": {
"sym": False,
"group_size": 128,
@ -385,14 +429,71 @@ compression.
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
/home/ea/work/genai_env/lib/python3.8/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2024-04-19 10:35:50.012050: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-19 10:35:50.025002: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-19 10:35:50.060073: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-19 10:35:50.060108: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-19 10:35:50.060134: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-04-19 10:35:50.068691: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-19 10:35:50.069448: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-19 10:35:51.045741: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
Framework not specified. Using pt to export the model.
.. parsed-literal::
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
.. parsed-literal::
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Using framework PyTorch: 2.2.2+cpu
Overriding 1 configuration item(s)
- use_cache -> True
/home/ea/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
The cos_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
The sin_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
/home/ea/miniconda3/lib/python3.11/site-packages/optimum/exporters/openvino/model_patcher.py:311: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if sequence_length != 1:
.. parsed-literal::
Output()
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
.. raw:: html
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">
</pre>
.. parsed-literal::
Configuration saved in llama-3-8b-instruct/INT4_compressed_weights/openvino_config.json
Let’s compare model size for different compression types
@ -414,7 +515,7 @@ Let’s compare model size for different compression types
.. parsed-literal::
Size of model with INT4 compressed weights is 1734.02 MB
Size of model with INT4 compressed weights is 4435.75 MB
Select device for inference and model variant
@ -500,7 +601,7 @@ Select device for inference and model variant
.. parsed-literal::
Loading model from phi-2/INT4_compressed_weights
Loading model from llama-3-8b-instruct/INT4_compressed_weights
.. parsed-literal::
@ -938,3 +1039,17 @@ generation parameters:
# If you are launching remotely, specify server_name and server_port
# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`
# To learn more please refer to the Gradio docs: https://gradio.app/docs/
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.

Some files were not shown because too many files have changed in this diff Show More