+
+
+
+.. parsed-literal::
+
+ INFO:nncf:3 ignored nodes were found by types in the NNCFGraph
+ INFO:nncf:182 ignored nodes were found by name in the NNCFGraph
+ INFO:nncf:Not adding activation input quantizer for operation: 37 __module.transformer.embed.conv/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 2883 __module.transformer.mlm_layer.conv1/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 3243 __module.transformer.mlm_layer.conv2/aten::_convolution/Convolution
+
+
+
+.. parsed-literal::
+
+ Output()
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
+ return Tensor(self.data * unwrap_tensor_data(other))
+
+
+
+.. raw:: html
+
+
+
+
+
+
Demo generation with quantized pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: ipython3
%%skip not $to_quantize.value
-
+
original_ov_transformer_model = pipe.transformer.transformer
pipe.transformer.transformer = core.compile_model(QUANTIZED_TRANSFORMER_OV_PATH, device.value)
-
+
image = pipe(prompt, generator=torch.Generator('cpu').manual_seed(8)).images[0]
image.save('text2image_256_quantized.png')
-
+
pipe.transformer.transformer = original_ov_transformer_model
-
+
display(image)
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/configuration_utils.py:139: FutureWarning: Accessing config attribute `_execution_device` directly via 'AmusedPipeline' object attribute is deprecated. Please access '_execution_device' over 'AmusedPipeline's config object instead, e.g. 'scheduler.config._execution_device'.
+ deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
+
+
+
+.. parsed-literal::
+
+ 0%| | 0/12 [00:00 0:
images_batch = torch.stack(images[-batch_size:])
images = images[:-batch_size]
inception_score.update(images_batch)
kl_mean, kl_std = inception_score.compute()
-
+
pipe.transformer.transformer = original_ov_transformer_model
disable_progress_bar(pipe, disable=False)
-
+
return kl_mean, mean_perf_time
-
-
+
+
original_inception_score, original_time = compute_inception_score(TRANSFORMER_OV_PATH, VALIDATION_DATASET_SIZE)
print(f"Original pipeline Inception Score: {original_inception_score}")
quantized_inception_score, quantized_time = compute_inception_score(QUANTIZED_TRANSFORMER_OV_PATH, VALIDATION_DATASET_SIZE)
print(f"Quantized pipeline Inception Score: {quantized_inception_score}")
print(f"Quantization speed-up: {original_time / quantized_time:.2f}x")
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: Metric `InceptionScore` will save all extracted features in buffer. For large datasets this may lead to large memory footprint.
+ warnings.warn(\*args, \*\*kwargs) # noqa: B028
+
+
+
+.. parsed-literal::
+
+ Computing Inception Score: 0%| | 0/100 [00:00
-
+
Your browser does not support the video tag.
diff --git a/docs/notebooks/async-api-with-output.rst b/docs/notebooks/async-api-with-output.rst
deleted file mode 100644
index 19a832b6964..00000000000
--- a/docs/notebooks/async-api-with-output.rst
+++ /dev/null
@@ -1,791 +0,0 @@
-Asynchronous Inference with OpenVINO™
-=====================================
-
-This notebook demonstrates how to use the `Async
-API `__
-for asynchronous execution with OpenVINO.
-
-OpenVINO Runtime supports inference in either synchronous or
-asynchronous mode. The key advantage of the Async API is that when a
-device is busy with inference, the application can perform other tasks
-in parallel (for example, populating inputs or scheduling other
-requests) rather than wait for the current inference to complete first.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Imports <#imports>`__
-- `Prepare model and data
- processing <#prepare-model-and-data-processing>`__
-
- - `Download test model <#download-test-model>`__
- - `Load the model <#load-the-model>`__
- - `Create functions for data
- processing <#create-functions-for-data-processing>`__
- - `Get the test video <#get-the-test-video>`__
-
-- `How to improve the throughput of video
- processing <#how-to-improve-the-throughput-of-video-processing>`__
-
- - `Sync Mode (default) <#sync-mode-default>`__
- - `Test performance in Sync Mode <#test-performance-in-sync-mode>`__
- - `Async Mode <#async-mode>`__
- - `Test the performance in Async
- Mode <#test-the-performance-in-async-mode>`__
- - `Compare the performance <#compare-the-performance>`__
-
-- `AsyncInferQueue <#asyncinferqueue>`__
-
- - `Setting Callback <#setting-callback>`__
- - `Test the performance with
- AsyncInferQueue <#test-the-performance-with-asyncinferqueue>`__
-
-Imports
--------
-
-
-
-.. code:: ipython3
-
- import platform
-
- %pip install -q "openvino>=2023.1.0"
- %pip install -q opencv-python
- if platform.system() != "windows":
- %pip install -q "matplotlib>=3.4"
- else:
- %pip install -q "matplotlib>=3.4,<3.7"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- import cv2
- import time
- import numpy as np
- import openvino as ov
- from IPython import display
- import matplotlib.pyplot as plt
-
- # Fetch the notebook utils script from the openvino_notebooks repo
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
- open("notebook_utils.py", "w").write(r.text)
-
- import notebook_utils as utils
-
-Prepare model and data processing
----------------------------------
-
-
-
-Download test model
-~~~~~~~~~~~~~~~~~~~
-
-
-
-We use a pre-trained model from OpenVINO’s `Open Model
-Zoo `__
-to start the test. In this case, the model will be executed to detect
-the person in each frame of the video.
-
-.. code:: ipython3
-
- # directory where model will be downloaded
- base_model_dir = "model"
-
- # model name as named in Open Model Zoo
- model_name = "person-detection-0202"
- precision = "FP16"
- model_path = f"model/intel/{model_name}/{precision}/{model_name}.xml"
- download_command = f"omz_downloader " f"--name {model_name} " f"--precision {precision} " f"--output_dir {base_model_dir} " f"--cache_dir {base_model_dir}"
- ! $download_command
-
-
-.. parsed-literal::
-
- ################|| Downloading person-detection-0202 ||################
-
- ========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.xml
-
-
-.. parsed-literal::
-
- ... 12%, 32 KB, 998 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 25%, 64 KB, 1010 KB/s, 0 seconds passed
-... 38%, 96 KB, 1446 KB/s, 0 seconds passed
-... 51%, 128 KB, 1336 KB/s, 0 seconds passed
-... 64%, 160 KB, 1638 KB/s, 0 seconds passed
-... 77%, 192 KB, 1927 KB/s, 0 seconds passed
-... 89%, 224 KB, 2206 KB/s, 0 seconds passed
-... 100%, 248 KB, 2423 KB/s, 0 seconds passed
-
-
- ========== Downloading model/intel/person-detection-0202/FP16/person-detection-0202.bin
-
-
-.. parsed-literal::
-
- ... 0%, 32 KB, 646 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 1%, 64 KB, 991 KB/s, 0 seconds passed
-... 2%, 96 KB, 1232 KB/s, 0 seconds passed
-... 3%, 128 KB, 1330 KB/s, 0 seconds passed
-... 4%, 160 KB, 1640 KB/s, 0 seconds passed
-... 5%, 192 KB, 1860 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 6%, 224 KB, 2064 KB/s, 0 seconds passed
-... 7%, 256 KB, 2271 KB/s, 0 seconds passed
-... 8%, 288 KB, 2239 KB/s, 0 seconds passed
-... 9%, 320 KB, 2476 KB/s, 0 seconds passed
-... 9%, 352 KB, 2698 KB/s, 0 seconds passed
-... 10%, 384 KB, 2906 KB/s, 0 seconds passed
-... 11%, 416 KB, 3122 KB/s, 0 seconds passed
-... 12%, 448 KB, 3335 KB/s, 0 seconds passed
-... 13%, 480 KB, 3537 KB/s, 0 seconds passed
-... 14%, 512 KB, 3723 KB/s, 0 seconds passed
-... 15%, 544 KB, 3847 KB/s, 0 seconds passed
-... 16%, 576 KB, 4016 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 17%, 608 KB, 3756 KB/s, 0 seconds passed
-... 18%, 640 KB, 3938 KB/s, 0 seconds passed
-... 18%, 672 KB, 4121 KB/s, 0 seconds passed
-... 19%, 704 KB, 4300 KB/s, 0 seconds passed
-... 20%, 736 KB, 4484 KB/s, 0 seconds passed
-... 21%, 768 KB, 4666 KB/s, 0 seconds passed
-... 22%, 800 KB, 4751 KB/s, 0 seconds passed
-... 23%, 832 KB, 4819 KB/s, 0 seconds passed
-... 24%, 864 KB, 4944 KB/s, 0 seconds passed
-... 25%, 896 KB, 5048 KB/s, 0 seconds passed
-... 26%, 928 KB, 5115 KB/s, 0 seconds passed
-... 27%, 960 KB, 5202 KB/s, 0 seconds passed
-... 27%, 992 KB, 5285 KB/s, 0 seconds passed
-... 28%, 1024 KB, 5369 KB/s, 0 seconds passed
-... 29%, 1056 KB, 5452 KB/s, 0 seconds passed
-... 30%, 1088 KB, 5568 KB/s, 0 seconds passed
-... 31%, 1120 KB, 5674 KB/s, 0 seconds passed
-... 32%, 1152 KB, 5773 KB/s, 0 seconds passed
-... 33%, 1184 KB, 5877 KB/s, 0 seconds passed
-... 34%, 1216 KB, 5951 KB/s, 0 seconds passed
-... 35%, 1248 KB, 6041 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 36%, 1280 KB, 6132 KB/s, 0 seconds passed
-... 36%, 1312 KB, 6225 KB/s, 0 seconds passed
-... 37%, 1344 KB, 6354 KB/s, 0 seconds passed
-... 38%, 1376 KB, 6446 KB/s, 0 seconds passed
-... 39%, 1408 KB, 6522 KB/s, 0 seconds passed
-... 40%, 1440 KB, 6650 KB/s, 0 seconds passed
-... 41%, 1472 KB, 6739 KB/s, 0 seconds passed
-... 42%, 1504 KB, 6827 KB/s, 0 seconds passed
-... 43%, 1536 KB, 6935 KB/s, 0 seconds passed
-... 44%, 1568 KB, 7017 KB/s, 0 seconds passed
-... 45%, 1600 KB, 7124 KB/s, 0 seconds passed
-... 45%, 1632 KB, 7201 KB/s, 0 seconds passed
-... 46%, 1664 KB, 7306 KB/s, 0 seconds passed
-... 47%, 1696 KB, 7411 KB/s, 0 seconds passed
-... 48%, 1728 KB, 7526 KB/s, 0 seconds passed
-... 49%, 1760 KB, 7611 KB/s, 0 seconds passed
-... 50%, 1792 KB, 7697 KB/s, 0 seconds passed
-... 51%, 1824 KB, 7816 KB/s, 0 seconds passed
-... 52%, 1856 KB, 7912 KB/s, 0 seconds passed
-... 53%, 1888 KB, 8016 KB/s, 0 seconds passed
-... 54%, 1920 KB, 8115 KB/s, 0 seconds passed
-... 54%, 1952 KB, 8215 KB/s, 0 seconds passed
-... 55%, 1984 KB, 8313 KB/s, 0 seconds passed
-... 56%, 2016 KB, 8409 KB/s, 0 seconds passed
-... 57%, 2048 KB, 8507 KB/s, 0 seconds passed
-... 58%, 2080 KB, 8608 KB/s, 0 seconds passed
-... 59%, 2112 KB, 8706 KB/s, 0 seconds passed
-... 60%, 2144 KB, 8801 KB/s, 0 seconds passed
-... 61%, 2176 KB, 8893 KB/s, 0 seconds passed
-... 62%, 2208 KB, 8987 KB/s, 0 seconds passed
-... 63%, 2240 KB, 9085 KB/s, 0 seconds passed
-... 64%, 2272 KB, 9202 KB/s, 0 seconds passed
-... 64%, 2304 KB, 9302 KB/s, 0 seconds passed
-... 65%, 2336 KB, 9396 KB/s, 0 seconds passed
-... 66%, 2368 KB, 9489 KB/s, 0 seconds passed
-... 67%, 2400 KB, 9579 KB/s, 0 seconds passed
-... 68%, 2432 KB, 9694 KB/s, 0 seconds passed
-... 69%, 2464 KB, 9791 KB/s, 0 seconds passed
-... 70%, 2496 KB, 9879 KB/s, 0 seconds passed
-... 71%, 2528 KB, 9993 KB/s, 0 seconds passed
-... 72%, 2560 KB, 10088 KB/s, 0 seconds passed
-... 73%, 2592 KB, 10186 KB/s, 0 seconds passed
-... 73%, 2624 KB, 10299 KB/s, 0 seconds passed
-... 74%, 2656 KB, 10376 KB/s, 0 seconds passed
-... 75%, 2688 KB, 10472 KB/s, 0 seconds passed
-... 76%, 2720 KB, 10566 KB/s, 0 seconds passed
-... 77%, 2752 KB, 10673 KB/s, 0 seconds passed
-
-.. parsed-literal::
-
- ... 78%, 2784 KB, 10764 KB/s, 0 seconds passed
-... 79%, 2816 KB, 10874 KB/s, 0 seconds passed
-... 80%, 2848 KB, 10964 KB/s, 0 seconds passed
-... 81%, 2880 KB, 11062 KB/s, 0 seconds passed
-... 82%, 2912 KB, 11172 KB/s, 0 seconds passed
-... 82%, 2944 KB, 11241 KB/s, 0 seconds passed
-... 83%, 2976 KB, 11346 KB/s, 0 seconds passed
-... 84%, 3008 KB, 11451 KB/s, 0 seconds passed
-... 85%, 3040 KB, 11528 KB/s, 0 seconds passed
-... 86%, 3072 KB, 11634 KB/s, 0 seconds passed
-... 87%, 3104 KB, 11720 KB/s, 0 seconds passed
-... 88%, 3136 KB, 11817 KB/s, 0 seconds passed
-... 89%, 3168 KB, 11921 KB/s, 0 seconds passed
-... 90%, 3200 KB, 12009 KB/s, 0 seconds passed
-... 91%, 3232 KB, 12114 KB/s, 0 seconds passed
-... 91%, 3264 KB, 12197 KB/s, 0 seconds passed
-... 92%, 3296 KB, 12274 KB/s, 0 seconds passed
-... 93%, 3328 KB, 12377 KB/s, 0 seconds passed
-... 94%, 3360 KB, 12462 KB/s, 0 seconds passed
-... 95%, 3392 KB, 12563 KB/s, 0 seconds passed
-... 96%, 3424 KB, 12652 KB/s, 0 seconds passed
-... 97%, 3456 KB, 12756 KB/s, 0 seconds passed
-... 98%, 3488 KB, 12845 KB/s, 0 seconds passed
-... 99%, 3520 KB, 12947 KB/s, 0 seconds passed
-... 100%, 3549 KB, 13037 KB/s, 0 seconds passed
-
-
-
-
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- core = ov.Core()
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="CPU",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
-
-
-
-Load the model
-~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- # initialize OpenVINO runtime
- core = ov.Core()
-
- # read the network and corresponding weights from file
- model = core.read_model(model=model_path)
-
- # compile the model for the CPU (you can choose manually CPU, GPU etc.)
- # or let the engine choose the best available device (AUTO)
- compiled_model = core.compile_model(model=model, device_name=device.value)
-
- # get input node
- input_layer_ir = model.input(0)
- N, C, H, W = input_layer_ir.shape
- shape = (H, W)
-
-Create functions for data processing
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- def preprocess(image):
- """
- Define the preprocess function for input data
-
- :param: image: the orignal input frame
- :returns:
- resized_image: the image processed
- """
- resized_image = cv2.resize(image, shape)
- resized_image = cv2.cvtColor(np.array(resized_image), cv2.COLOR_BGR2RGB)
- resized_image = resized_image.transpose((2, 0, 1))
- resized_image = np.expand_dims(resized_image, axis=0).astype(np.float32)
- return resized_image
-
-
- def postprocess(result, image, fps):
- """
- Define the postprocess function for output data
-
- :param: result: the inference results
- image: the orignal input frame
- fps: average throughput calculated for each frame
- :returns:
- image: the image with bounding box and fps message
- """
- detections = result.reshape(-1, 7)
- for i, detection in enumerate(detections):
- _, image_id, confidence, xmin, ymin, xmax, ymax = detection
- if confidence > 0.5:
- xmin = int(max((xmin * image.shape[1]), 10))
- ymin = int(max((ymin * image.shape[0]), 10))
- xmax = int(min((xmax * image.shape[1]), image.shape[1] - 10))
- ymax = int(min((ymax * image.shape[0]), image.shape[0] - 10))
- cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (0, 255, 0), 2)
- cv2.putText(
- image,
- str(round(fps, 2)) + " fps",
- (5, 20),
- cv2.FONT_HERSHEY_SIMPLEX,
- 0.7,
- (0, 255, 0),
- 3,
- )
- return image
-
-Get the test video
-~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- video_path = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/CEO%20Pat%20Gelsinger%20on%20Leading%20Intel.mp4"
-
-How to improve the throughput of video processing
--------------------------------------------------
-
-
-
-Below, we compare the performance of the synchronous and async-based
-approaches:
-
-Sync Mode (default)
-~~~~~~~~~~~~~~~~~~~
-
-
-
-Let us see how video processing works with the default approach. Using
-the synchronous approach, the frame is captured with OpenCV and then
-immediately processed:
-
-.. figure:: https://user-images.githubusercontent.com/91237924/168452573-d354ea5b-7966-44e5-813d-f9053be4338a.png
- :alt: drawing
-
- drawing
-
-::
-
- while(true) {
- // capture frame
- // populate CURRENT InferRequest
- // Infer CURRENT InferRequest
- //this call is synchronous
- // display CURRENT result
- }
-
-\``\`
-
-.. code:: ipython3
-
- def sync_api(source, flip, fps, use_popup, skip_first_frames):
- """
- Define the main function for video processing in sync mode
-
- :param: source: the video path or the ID of your webcam
- :returns:
- sync_fps: the inference throughput in sync mode
- """
- frame_number = 0
- infer_request = compiled_model.create_infer_request()
- player = None
- try:
- # Create a video player
- player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
- # Start capturing
- start_time = time.time()
- player.start()
- if use_popup:
- title = "Press ESC to Exit"
- cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)
- while True:
- frame = player.next()
- if frame is None:
- print("Source ended")
- break
- resized_frame = preprocess(frame)
- infer_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
- # Start the inference request in synchronous mode
- infer_request.infer()
- res = infer_request.get_output_tensor(0).data
- stop_time = time.time()
- total_time = stop_time - start_time
- frame_number = frame_number + 1
- sync_fps = frame_number / total_time
- frame = postprocess(res, frame, sync_fps)
- # Display the results
- if use_popup:
- cv2.imshow(title, frame)
- key = cv2.waitKey(1)
- # escape = 27
- if key == 27:
- break
- else:
- # Encode numpy array to jpg
- _, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
- # Create IPython image
- i = display.Image(data=encoded_img)
- # Display the image in this notebook
- display.clear_output(wait=True)
- display.display(i)
- # ctrl-c
- except KeyboardInterrupt:
- print("Interrupted")
- # Any different error
- except RuntimeError as e:
- print(e)
- finally:
- if use_popup:
- cv2.destroyAllWindows()
- if player is not None:
- # stop capturing
- player.stop()
- return sync_fps
-
-Test performance in Sync Mode
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- sync_fps = sync_api(source=video_path, flip=False, fps=30, use_popup=False, skip_first_frames=800)
- print(f"average throuput in sync mode: {sync_fps:.2f} fps")
-
-
-
-.. image:: async-api-with-output_files/async-api-with-output_17_0.png
-
-
-.. parsed-literal::
-
- Source ended
- average throuput in sync mode: 44.01 fps
-
-
-Async Mode
-~~~~~~~~~~
-
-
-
-Let us see how the OpenVINO Async API can improve the overall frame rate
-of an application. The key advantage of the Async approach is as
-follows: while a device is busy with the inference, the application can
-do other things in parallel (for example, populating inputs or
-scheduling other requests) rather than wait for the current inference to
-complete first.
-
-.. figure:: https://user-images.githubusercontent.com/91237924/168452572-c2ff1c59-d470-4b85-b1f6-b6e1dac9540e.png
- :alt: drawing
-
- drawing
-
-In the example below, inference is applied to the results of the video
-decoding. So it is possible to keep multiple infer requests, and while
-the current request is processed, the input frame for the next is being
-captured. This essentially hides the latency of capturing, so that the
-overall frame rate is rather determined only by the slowest part of the
-pipeline (decoding vs inference) and not by the sum of the stages.
-
-::
-
- while(true) {
- // capture frame
- // populate NEXT InferRequest
- // start NEXT InferRequest
- // this call is async and returns immediately
- // wait for the CURRENT InferRequest
- // display CURRENT result
- // swap CURRENT and NEXT InferRequests
- }
-
-.. code:: ipython3
-
- def async_api(source, flip, fps, use_popup, skip_first_frames):
- """
- Define the main function for video processing in async mode
-
- :param: source: the video path or the ID of your webcam
- :returns:
- async_fps: the inference throughput in async mode
- """
- frame_number = 0
- # Create 2 infer requests
- curr_request = compiled_model.create_infer_request()
- next_request = compiled_model.create_infer_request()
- player = None
- async_fps = 0
- try:
- # Create a video player
- player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
- # Start capturing
- start_time = time.time()
- player.start()
- if use_popup:
- title = "Press ESC to Exit"
- cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)
- # Capture CURRENT frame
- frame = player.next()
- resized_frame = preprocess(frame)
- curr_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
- # Start the CURRENT inference request
- curr_request.start_async()
- while True:
- # Capture NEXT frame
- next_frame = player.next()
- if next_frame is None:
- print("Source ended")
- break
- resized_frame = preprocess(next_frame)
- next_request.set_tensor(input_layer_ir, ov.Tensor(resized_frame))
- # Start the NEXT inference request
- next_request.start_async()
- # Waiting for CURRENT inference result
- curr_request.wait()
- res = curr_request.get_output_tensor(0).data
- stop_time = time.time()
- total_time = stop_time - start_time
- frame_number = frame_number + 1
- async_fps = frame_number / total_time
- frame = postprocess(res, frame, async_fps)
- # Display the results
- if use_popup:
- cv2.imshow(title, frame)
- key = cv2.waitKey(1)
- # escape = 27
- if key == 27:
- break
- else:
- # Encode numpy array to jpg
- _, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
- # Create IPython image
- i = display.Image(data=encoded_img)
- # Display the image in this notebook
- display.clear_output(wait=True)
- display.display(i)
- # Swap CURRENT and NEXT frames
- frame = next_frame
- # Swap CURRENT and NEXT infer requests
- curr_request, next_request = next_request, curr_request
- # ctrl-c
- except KeyboardInterrupt:
- print("Interrupted")
- # Any different error
- except RuntimeError as e:
- print(e)
- finally:
- if use_popup:
- cv2.destroyAllWindows()
- if player is not None:
- # stop capturing
- player.stop()
- return async_fps
-
-Test the performance in Async Mode
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- async_fps = async_api(source=video_path, flip=False, fps=30, use_popup=False, skip_first_frames=800)
- print(f"average throuput in async mode: {async_fps:.2f} fps")
-
-
-
-.. image:: async-api-with-output_files/async-api-with-output_21_0.png
-
-
-.. parsed-literal::
-
- Source ended
- average throuput in async mode: 73.74 fps
-
-
-Compare the performance
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- width = 0.4
- fontsize = 14
-
- plt.rc("font", size=fontsize)
- fig, ax = plt.subplots(1, 1, figsize=(10, 8))
-
- rects1 = ax.bar([0], sync_fps, width, color="#557f2d")
- rects2 = ax.bar([width], async_fps, width)
- ax.set_ylabel("frames per second")
- ax.set_xticks([0, width])
- ax.set_xticklabels(["Sync mode", "Async mode"])
- ax.set_xlabel("Higher is better")
-
- fig.suptitle("Sync mode VS Async mode")
- fig.tight_layout()
-
- plt.show()
-
-
-
-.. image:: async-api-with-output_files/async-api-with-output_23_0.png
-
-
-``AsyncInferQueue``
--------------------
-
-
-
-Asynchronous mode pipelines can be supported with the
-`AsyncInferQueue `__
-wrapper class. This class automatically spawns the pool of
-``InferRequest`` objects (also called “jobs”) and provides
-synchronization mechanisms to control the flow of the pipeline. It is a
-simpler way to manage the infer request queue in Asynchronous mode.
-
-Setting Callback
-~~~~~~~~~~~~~~~~
-
-
-
-When ``callback`` is set, any job that ends inference calls upon the
-Python function. The ``callback`` function must have two arguments: one
-is the request that calls the ``callback``, which provides the
-``InferRequest`` API; the other is called “user data”, which provides
-the possibility of passing runtime values.
-
-.. code:: ipython3
-
- def callback(infer_request, info) -> None:
- """
- Define the callback function for postprocessing
-
- :param: infer_request: the infer_request object
- info: a tuple includes original frame and starts time
- :returns:
- None
- """
- global frame_number
- global total_time
- global inferqueue_fps
- stop_time = time.time()
- frame, start_time = info
- total_time = stop_time - start_time
- frame_number = frame_number + 1
- inferqueue_fps = frame_number / total_time
-
- res = infer_request.get_output_tensor(0).data[0]
- frame = postprocess(res, frame, inferqueue_fps)
- # Encode numpy array to jpg
- _, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
- # Create IPython image
- i = display.Image(data=encoded_img)
- # Display the image in this notebook
- display.clear_output(wait=True)
- display.display(i)
-
-.. code:: ipython3
-
- def inferqueue(source, flip, fps, skip_first_frames) -> None:
- """
- Define the main function for video processing with async infer queue
-
- :param: source: the video path or the ID of your webcam
- :retuns:
- None
- """
- # Create infer requests queue
- infer_queue = ov.AsyncInferQueue(compiled_model, 2)
- infer_queue.set_callback(callback)
- player = None
- try:
- # Create a video player
- player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)
- # Start capturing
- start_time = time.time()
- player.start()
- while True:
- # Capture frame
- frame = player.next()
- if frame is None:
- print("Source ended")
- break
- resized_frame = preprocess(frame)
- # Start the inference request with async infer queue
- infer_queue.start_async({input_layer_ir.any_name: resized_frame}, (frame, start_time))
- except KeyboardInterrupt:
- print("Interrupted")
- # Any different error
- except RuntimeError as e:
- print(e)
- finally:
- infer_queue.wait_all()
- player.stop()
-
-Test the performance with ``AsyncInferQueue``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- frame_number = 0
- total_time = 0
- inferqueue(source=video_path, flip=False, fps=30, skip_first_frames=800)
- print(f"average throughput in async mode with async infer queue: {inferqueue_fps:.2f} fps")
-
-
-
-.. image:: async-api-with-output_files/async-api-with-output_29_0.png
-
-
-.. parsed-literal::
-
- average throughput in async mode with async infer queue: 112.89 fps
-
diff --git a/docs/notebooks/async-api-with-output_files/async-api-with-output_23_0.png b/docs/notebooks/async-api-with-output_files/async-api-with-output_23_0.png
index b7a2dc965c6..2aad811bb6b 100644
--- a/docs/notebooks/async-api-with-output_files/async-api-with-output_23_0.png
+++ b/docs/notebooks/async-api-with-output_files/async-api-with-output_23_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:3ee108fdee6cf1a56efa4a2347f0bc0c4cd09b6b30b121d114cfdd2cd87a9152
-size 30443
+oid sha256:deee8ff5a3fba807c811d63eeb7f516cbdc82d47f6b4c4ae797742ff6327a54d
+size 30483
diff --git a/docs/notebooks/auto-device-with-output.rst b/docs/notebooks/auto-device-with-output.rst
index bfe8b33f916..6790703a654 100644
--- a/docs/notebooks/auto-device-with-output.rst
+++ b/docs/notebooks/auto-device-with-output.rst
@@ -82,10 +82,6 @@ Import modules and create Core
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -190,15 +186,15 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
- [23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:LOG_LEVEL=LOG_INFO
- [23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
- [23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
- [23:04:35.7467]I[plugin.cpp:418][AUTO] device:CPU, config:PERF_COUNT=NO
- [23:04:35.7467]I[plugin.cpp:423][AUTO] device:CPU, priority:0
- [23:04:35.7467]I[schedule.cpp:17][AUTO] scheduler starting
- [23:04:35.7468]I[auto_schedule.cpp:131][AUTO] select device:CPU
- [23:04:35.9061]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
- [23:04:35.9063]I[plugin.cpp:451][AUTO] underlying hardware does not support hardware context
+ [23:25:36.8706]I[plugin.cpp:418][AUTO] device:CPU, config:LOG_LEVEL=LOG_INFO
+ [23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT=LATENCY
+ [23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERFORMANCE_HINT_NUM_REQUESTS=0
+ [23:25:36.8707]I[plugin.cpp:418][AUTO] device:CPU, config:PERF_COUNT=NO
+ [23:25:36.8707]I[plugin.cpp:423][AUTO] device:CPU, priority:0
+ [23:25:36.8707]I[schedule.cpp:17][AUTO] scheduler starting
+ [23:25:36.8707]I[auto_schedule.cpp:131][AUTO] select device:CPU
+ [23:25:37.0101]I[auto_schedule.cpp:109][AUTO] device:CPU compiling model finished
+ [23:25:37.0103]I[plugin.cpp:451][AUTO] underlying hardware does not support hardware context
Successfully compiled model without a device_name.
@@ -212,7 +208,7 @@ By default, ``compile_model`` API will select **AUTO** as
.. parsed-literal::
Deleted compiled_model
- [23:04:35.9172]I[schedule.cpp:303][AUTO] scheduler ending
+ [23:25:37.0205]I[schedule.cpp:303][AUTO] scheduler ending
Explicitly pass AUTO as device_name to Core::compile_model API
@@ -370,7 +366,7 @@ executed on CPU until GPU is ready.
.. parsed-literal::
- Time to load model using AUTO device and get first inference: 0.15 seconds.
+ Time to load model using AUTO device and get first inference: 0.19 seconds.
.. code:: ipython3
@@ -541,45 +537,13 @@ Loop for inference and update the FPS/Latency every
.. parsed-literal::
Compiling Model for AUTO device with THROUGHPUT hint
-
-
-.. parsed-literal::
-
Start inference, 6 groups of FPS/latency will be measured over 10s intervals
-
-
-.. parsed-literal::
-
- throughput: 179.12fps, latency: 31.83ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 181.09fps, latency: 32.33ms, time interval: 10.01s
-
-
-.. parsed-literal::
-
- throughput: 179.44fps, latency: 32.62ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 179.98fps, latency: 32.57ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 179.55fps, latency: 32.61ms, time interval: 10.01s
-
-
-.. parsed-literal::
-
- throughput: 179.60fps, latency: 32.58ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
+ throughput: 177.50fps, latency: 32.10ms, time interval: 10.00s
+ throughput: 179.46fps, latency: 32.64ms, time interval: 10.01s
+ throughput: 179.28fps, latency: 32.70ms, time interval: 10.00s
+ throughput: 177.92fps, latency: 32.86ms, time interval: 10.01s
+ throughput: 178.98fps, latency: 32.68ms, time interval: 10.02s
+ throughput: 178.91fps, latency: 32.77ms, time interval: 10.01s
Done
@@ -624,45 +588,13 @@ Loop for inference and update the FPS/Latency for each
.. parsed-literal::
Compiling Model for AUTO Device with LATENCY hint
-
-
-.. parsed-literal::
-
Start inference, 6 groups fps/latency will be out with 10s interval
-
-
-.. parsed-literal::
-
- throughput: 137.86fps, latency: 6.72ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 140.86fps, latency: 6.72ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 140.85fps, latency: 6.72ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 140.28fps, latency: 6.69ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 140.66fps, latency: 6.70ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
- throughput: 140.48fps, latency: 6.68ms, time interval: 10.00s
-
-
-.. parsed-literal::
-
+ throughput: 135.86fps, latency: 6.81ms, time interval: 10.00s
+ throughput: 138.93fps, latency: 6.82ms, time interval: 10.01s
+ throughput: 138.89fps, latency: 6.82ms, time interval: 10.00s
+ throughput: 138.82fps, latency: 6.81ms, time interval: 10.01s
+ throughput: 138.99fps, latency: 6.82ms, time interval: 10.00s
+ throughput: 139.01fps, latency: 6.82ms, time interval: 10.01s
Done
diff --git a/docs/notebooks/auto-device-with-output_files/auto-device-with-output_27_0.png b/docs/notebooks/auto-device-with-output_files/auto-device-with-output_27_0.png
index a5e6e606528..4b5d482c7b3 100644
--- a/docs/notebooks/auto-device-with-output_files/auto-device-with-output_27_0.png
+++ b/docs/notebooks/auto-device-with-output_files/auto-device-with-output_27_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:84c73fa64ec907ad0be08db15bebc96b35c8b26b29d29f657f8ceeb75aab5662
-size 27580
+oid sha256:2ce598bcda980dc39d1fc49884a02509af4d0f599c4dd674e07b994af39cf533
+size 27041
diff --git a/docs/notebooks/auto-device-with-output_files/auto-device-with-output_28_0.png b/docs/notebooks/auto-device-with-output_files/auto-device-with-output_28_0.png
index 97eef2a1702..31d8b7b761c 100644
--- a/docs/notebooks/auto-device-with-output_files/auto-device-with-output_28_0.png
+++ b/docs/notebooks/auto-device-with-output_files/auto-device-with-output_28_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:61109ff91028e7d3caeed1fd125fd850785bb6341898e0dce5a87aece1166832
-size 39983
+oid sha256:1efc7ab0c2433842a374eb59dc1be264bb613039696dbdceb5b51c97c8ad666b
+size 39972
diff --git a/docs/notebooks/blip-diffusion-subject-generation-with-output.rst b/docs/notebooks/blip-diffusion-subject-generation-with-output.rst
deleted file mode 100644
index 8a3d4beb770..00000000000
--- a/docs/notebooks/blip-diffusion-subject-generation-with-output.rst
+++ /dev/null
@@ -1,1554 +0,0 @@
-Subject-driven image generation and editing using BLIP Diffusion and OpenVINO
-=============================================================================
-
-|image0| `BLIP-Diffusion `__ is a
-text-to-image diffusion model with built-in support for multimodal
-subject-and-text condition. BLIP-Diffusion enables zero-shot
-subject-driven generation, and efficient fine-tuning for customized
-subjects with up to 20x speedup. In addition, BLIP-Diffusion can be
-flexibly combined with ControlNet and prompt-to-prompt to enable novel
-subject-driven generation and editing applications.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Prerequisites <#prerequisites>`__
-- `Load the model <#load-the-model>`__
-- `Infer the original model <#infer-the-original-model>`__
-
- - `Zero-Shot subject-driven
- generation <#zero-shot-subject-driven-generation>`__
- - `Controlled subject-driven generation
- (Canny-edge) <#controlled-subject-driven-generation-canny-edge>`__
- - `Controlled subject-driven generation
- (Scribble) <#controlled-subject-driven-generation-scribble>`__
-
-- `Convert the model to OpenVINO Intermediate Representation
- (IR) <#convert-the-model-to-openvino-intermediate-representation-ir>`__
-
- - `Q-Former <#q-former>`__
- - `Text encoder <#text-encoder>`__
- - `ControlNet <#controlnet>`__
- - `UNet <#unet>`__
- - `Variational Autoencoder (VAE) <#variational-autoencoder-vae>`__
- - `Select inference device <#select-inference-device>`__
-
-- `Inference <#inference>`__
-
- - `Zero-Shot subject-driven
- generation <#zero-shot-subject-driven-generation>`__
- - `Controlled subject-driven generation
- (Canny-edge) <#controlled-subject-driven-generation-canny-edge>`__
- - `Controlled subject-driven generation
- (Scribble) <#controlled-subject-driven-generation-scribble>`__
-
-- `Interactive inference <#interactive-inference>`__
-
-.. |image0| image:: https://github.com/salesforce/LAVIS/raw/main/projects/blip-diffusion/teaser-website.png
-
-Prerequisites
--------------
-
-
-
-.. code:: ipython3
-
- import platform
-
- %pip install -q "openvino>=2023.1.0" Pillow "gradio>=4.19"
- %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "torch>=2.1.0" "transformers>=4.36" accelerate controlnet_aux "diffusers>=0.23.0" "peft==0.6.2"
-
- if platform.system() != "Windows":
- %pip install -q "matplotlib>=3.4"
- else:
- %pip install -q "matplotlib>=3.4,<3.7"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- from pathlib import Path
- import gc
- from typing import List, Optional, Union
- from functools import partial
-
- import diffusers
- import torch
- import matplotlib.pyplot as plt
- import PIL
- import numpy as np
- import gradio as gr
- import controlnet_aux
-
- import openvino as ov
-
-
-.. parsed-literal::
-
- /home/itrushkin/.virtualenvs/blip_diffusion/lib/python3.10/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
- warnings.warn(
-
-
-.. code:: ipython3
-
- MODELS_DIR = Path("models")
- QFORMER_PATH = MODELS_DIR / "qformer.xml"
- TEXT_ENCODER_PATH = MODELS_DIR / "text_encoder.xml"
- NEG_TEXT_ENCODER_PATH = MODELS_DIR / "neg_text_encoder.xml"
- CONTROLNET_PATH = MODELS_DIR / "controlnet.xml"
- UNET_PATH = MODELS_DIR / "unet.xml"
- UNET_CONTROLNET_PATH = MODELS_DIR / "unet_controlnet.xml"
- VAE_PATH = MODELS_DIR / "vae.xml"
-
- DATA_DIR = Path("data")
- DOG_IMG_URL = "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/dog.jpg"
- DOG_IMG_PATH = DATA_DIR / "dog.jpg"
- KETTLE_IMG_URL = "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/kettle.jpg"
- KETTLE_IMG_PATH = DATA_DIR / "kettle.jpg"
- FLOWER_IMG_URL = "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/flower.jpg"
- FLOWER_IMG_PATH = DATA_DIR / "flower.jpg"
- BAG_IMG_URL = "https://huggingface.co/lllyasviel/sd-controlnet-scribble/resolve/main/images/bag.png"
- BAG_IMG_PATH = DATA_DIR / "bag.jpg"
-
- MODELS_DIR.mkdir(parents=True, exist_ok=True)
- DATA_DIR.mkdir(parents=True, exist_ok=True)
-
-Load the model
---------------
-
-
-
-We use Hugging Face ``diffusers`` library to load the model using
-``from_pretrained`` method.
-
-.. code:: ipython3
-
- pipe = diffusers.pipelines.BlipDiffusionPipeline.from_pretrained("ayushtues/blipdiffusion")
- pipe_controlnet = diffusers.pipelines.BlipDiffusionControlNetPipeline.from_pretrained("ayushtues/blipdiffusion-controlnet")
-
-
-.. parsed-literal::
-
- qformer/model.safetensors not found
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00`__ is a
-popular edge detection algorithm that produces high-quality edge maps
-from images.
-
-The approach is first to use the Canny edge detector to generate an edge
-map of the desired object. The edge map is then used to condition the
-diffusion model during image generation. This results in images that are
-more likely to contain the desired object and more faithful to the text
-description.
-
-.. code:: ipython3
-
- style_subject = ["flower"] # subject that defines the style
- tgt_subject = ["teapot"] # subject to generate.
- text_prompt = ["on a marble table"]
- cond_image = PIL.Image.open(KETTLE_IMG_PATH).resize((512, 512))
- canny = controlnet_aux.CannyDetector()
- cldm_cond_image = canny(cond_image, 30, 70, output_type="pil")
- cldm_cond_image = [cldm_cond_image]
-
- style_image = PIL.Image.open(FLOWER_IMG_PATH)
-
-
- guidance_scale = 7.5
- num_inference_steps = 50
- negative_prompt = "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, out of frame, ugly, bad anatomy, bad proportions, deformed, blurry, duplicate"
-
-.. code:: ipython3
-
- output = pipe_controlnet(
- text_prompt,
- style_image,
- cldm_cond_image,
- style_subject,
- tgt_subject,
- guidance_scale=guidance_scale,
- num_inference_steps=num_inference_steps,
- neg_prompt=negative_prompt,
- height=512,
- width=512,
- )
-
-
-
-.. parsed-literal::
-
- 0%| | 0/51 [00:00`__ (HED) is a deep
-learning model for edge detection.
-
-HED first uses the scribble to generate a seed map. The seed map is a
-binary image where the scribbled pixels are set to 1 and the other
-pixels are set to 0. Then, it uses the seed map to initialize a
-diffusion process. The diffusion process gradually spreads the edge
-information from the seed pixels to the other pixels in the image. The
-diffusion process is stopped when the edge map converges. The converged
-edge map is the final output of HED and input of our diffusion model.
-
-.. code:: ipython3
-
- style_subject = ["flower"] # subject that defines the style
- tgt_subject = ["bag"] # subject to generate.
- text_prompt = ["on a table"]
- bag_img = PIL.Image.open(BAG_IMG_PATH)
- cldm_cond_image = bag_img.resize((512, 512))
- hed = controlnet_aux.HEDdetector.from_pretrained("lllyasviel/Annotators")
- cldm_cond_image = hed(cldm_cond_image)
- cldm_cond_image = [cldm_cond_image]
-
- guidance_scale = 7.5
- num_inference_steps = 50
- negative_prompt = "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, out of frame, ugly, bad anatomy, bad proportions, deformed, blurry, duplicate"
-
- output = pipe_controlnet(
- text_prompt,
- style_image,
- cldm_cond_image,
- style_subject,
- tgt_subject,
- guidance_scale=guidance_scale,
- num_inference_steps=num_inference_steps,
- neg_prompt=negative_prompt,
- height=512,
- width=512,
- )
-
-
-
-.. parsed-literal::
-
- 0%| | 0/51 [00:00`__ paper and is a
-transformer that accepts a fixed number a learnable query tokens and an
-input text. It is used in BLIP Diffusion pipeline as a multimodal
-encoder for image-text alignment. The query tokens interact with text
-through self-attention layers, and interact with frozen image features
-through cross-attention layers, and produces text-aligned image features
-as output. The output is of the same dimension as the number of query
-tokens.
-
-Original QFormer model takes raw text as input, so we redefine the
-``forward`` function to accept tokenization result as ``input_ids`` and
-``attention_mask`` tensors.
-
-.. code:: ipython3
-
- class OVQFormer(torch.nn.Module):
- def __init__(self, qformer):
- super().__init__()
- self._qformer = qformer
-
- def __getattr__(self, name):
- if name == "_qformer":
- return super().__getattr__(name)
- return getattr(self._qformer, name)
-
- def forward(
- self,
- text_input_ids,
- text_attention_mask,
- image_input,
- ):
- batch_size = text_input_ids.shape[0]
- query_atts = torch.ones((batch_size, self.query_tokens.size()[1]), dtype=torch.long)
- attention_mask = torch.cat([query_atts, text_attention_mask], dim=1)
-
- output_attentions = self.config.output_attentions
- output_hidden_states = self.config.output_hidden_states
- return_dict = self.config.use_return_dict
-
- query_length = self.query_tokens.shape[1]
-
- embedding_output = self.embeddings(input_ids=text_input_ids, query_embeds=self.query_tokens)
-
- # embedding_output = self.layernorm(query_embeds)
- # embedding_output = self.dropout(embedding_output)
-
- input_shape = embedding_output.size()[:-1]
- batch_size, seq_length = input_shape
- device = embedding_output.device
-
- image_embeds_frozen = self.visual_encoder(image_input).last_hidden_state
- # image_embeds_frozen = torch.ones_like(image_embeds_frozen)
- encoder_hidden_states = image_embeds_frozen
-
- if attention_mask is None:
- attention_mask = torch.ones(((batch_size, seq_length)), device=device)
-
- # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
- # ourselves in which case we just need to make it broadcastable to all heads.
- extended_attention_mask = self.get_extended_attention_mask(attention_mask, input_shape, device)
-
- # If a 2D or 3D attention mask is provided for the cross-attention
- # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
- if encoder_hidden_states is not None:
- if isinstance(encoder_hidden_states, list):
- encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].size()
- else:
- (
- encoder_batch_size,
- encoder_sequence_length,
- _,
- ) = encoder_hidden_states.size()
- encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
- encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
- encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
- else:
- encoder_extended_attention_mask = None
-
- head_mask = [None] * self.config.qformer_config.num_hidden_layers
-
- encoder_outputs = self.encoder(
- embedding_output,
- attention_mask=extended_attention_mask,
- head_mask=head_mask,
- encoder_hidden_states=encoder_hidden_states,
- encoder_attention_mask=encoder_extended_attention_mask,
- output_attentions=output_attentions,
- output_hidden_states=output_hidden_states,
- return_dict=return_dict,
- query_length=query_length,
- )
- sequence_output = encoder_outputs[0]
- return self.proj_layer(sequence_output[:, :query_length, :])
-
-.. code:: ipython3
-
- serialize_openvino(
- OVQFormer(qformer),
- QFORMER_PATH,
- example_input={
- "image_input": torch.randn(1, 3, 16, 16),
- "text_input_ids": torch.zeros((1, 3), dtype=torch.int64),
- "text_attention_mask": torch.zeros((1, 3), dtype=torch.int64),
- },
- input={
- "image_input": ((1, 3, 224, 224),),
- "text_input_ids": ((1, ov.Dimension(3, 77)), np.int64),
- "text_attention_mask": ((1, ov.Dimension(3, 77)), np.int64),
- },
- )
-
- del qformer
- gc.collect()
-
-
-
-
-.. parsed-literal::
-
- 0
-
-
-
-Text encoder
-~~~~~~~~~~~~
-
-
-
-BLIP-Diffusion pipeline uses CLIP text encoder, the default encoder for
-Stable Diffusion-based models. The only difference is it allows for an
-extra input of “context embeddings”, which are the query embeddings used
-in Q-Former. They pass through the CLIP model, along with the text
-embeddings, and interact with them using self-attention.
-
-.. code:: ipython3
-
- serialize_openvino(
- text_encoder,
- TEXT_ENCODER_PATH,
- example_input={
- "input_ids": torch.zeros((1, 61), dtype=torch.int64),
- "ctx_embeddings": torch.zeros((1, 16, 768)),
- "ctx_begin_pos": torch.tensor([2]),
- },
- input={
- "input_ids": ((1, 61), np.int64),
- "ctx_embeddings": ((1, 16, 768),),
- "ctx_begin_pos": ((1),),
- },
- )
-
- # Convert 2nd instance for negative prompt encoding
- serialize_openvino(
- text_encoder,
- NEG_TEXT_ENCODER_PATH,
- example_input={
- "input_ids": torch.zeros((1, 77), dtype=torch.int64),
- },
- input={
- "input_ids": ((1, 77), np.int64),
- },
- )
-
- del text_encoder
- gc.collect()
-
-
-
-
-.. parsed-literal::
-
- 0
-
-
-
-ControlNet
-~~~~~~~~~~
-
-
-
-The ControlNet model was introduced in `Adding Conditional Control to
-Text-to-Image Diffusion
-Models `__. It provides a
-greater degree of control over text-to-image generation by conditioning
-the model on additional inputs such as edge maps, depth maps,
-segmentation maps, and keypoints for pose detection.
-
-.. code:: ipython3
-
- controlnet.forward = partial(controlnet.forward, return_dict=False)
- example_input = {
- "sample": torch.randn(2, 4, 64, 64),
- "timestep": torch.tensor(1),
- "encoder_hidden_states": torch.randn(2, 77, 768),
- "controlnet_cond": torch.randn(2, 3, 512, 512),
- }
- with torch.no_grad():
- down_block_res_samples, mid_block_res_sample = controlnet(**example_input)
- serialize_openvino(
- controlnet,
- CONTROLNET_PATH,
- example_input=example_input,
- input={
- "sample": ((2, 4, 64, 64)),
- "timestep": ((),),
- "encoder_hidden_states": ((2, 77, 768),),
- "controlnet_cond": ((2, 3, 512, 512)),
- },
- )
- del controlnet
- gc.collect()
-
-
-
-
-.. parsed-literal::
-
- 4463
-
-
-
-UNet
-~~~~
-
-
-
-The `UNet `__ model is one of
-the most important components of a diffusion system because it
-facilitates the actual diffusion process.
-
-.. code:: ipython3
-
- from typing import Tuple
-
- serialize_openvino(
- unet,
- UNET_PATH,
- example_input={
- "sample": torch.randn(2, 4, 32, 32),
- "timestep": torch.tensor(1),
- "encoder_hidden_states": torch.randn(2, 77, 768),
- },
- input={
- "sample": ((2, 4, unet_sample_size, unet_sample_size),),
- "timestep": ((),),
- "encoder_hidden_states": ((2, 77, 768),),
- },
- )
-
- dtype_mapping = {
- torch.float32: ov.Type.f32,
- torch.float64: ov.Type.f64,
- torch.int32: ov.Type.i32,
- torch.int64: ov.Type.i64,
- }
-
-
- class UnetWrapper(torch.nn.Module):
- def __init__(
- self,
- unet,
- sample_dtype=torch.float32,
- timestep_dtype=torch.int64,
- encoder_hidden_states=torch.float32,
- down_block_additional_residuals=torch.float32,
- mid_block_additional_residual=torch.float32,
- ):
- super().__init__()
- self.unet = unet
- self.sample_dtype = sample_dtype
- self.timestep_dtype = timestep_dtype
- self.encoder_hidden_states_dtype = encoder_hidden_states
- self.down_block_additional_residuals_dtype = down_block_additional_residuals
- self.mid_block_additional_residual_dtype = mid_block_additional_residual
-
- def forward(
- self,
- sample: torch.Tensor,
- timestep: torch.Tensor,
- encoder_hidden_states: torch.Tensor,
- down_block_additional_residuals: Tuple[torch.Tensor],
- mid_block_additional_residual: torch.Tensor,
- ):
- sample.to(self.sample_dtype)
- timestep.to(self.timestep_dtype)
- encoder_hidden_states.to(self.encoder_hidden_states_dtype)
- down_block_additional_residuals = [res.to(self.down_block_additional_residuals_dtype) for res in down_block_additional_residuals]
- mid_block_additional_residual.to(self.mid_block_additional_residual_dtype)
- return self.unet(
- sample,
- timestep,
- encoder_hidden_states,
- down_block_additional_residuals=down_block_additional_residuals,
- mid_block_additional_residual=mid_block_additional_residual,
- )
-
-
- def flatten_inputs(inputs):
- flat_inputs = []
- for input_data in inputs:
- if input_data is None:
- continue
- if isinstance(input_data, (list, tuple)):
- flat_inputs.extend(flatten_inputs(input_data))
- else:
- flat_inputs.append(input_data)
- return flat_inputs
-
-
- # convert 2nd time for stylization task
- example_input = {
- "sample": torch.randn(2, 4, unet_sample_size, unet_sample_size),
- "timestep": torch.tensor(1),
- "encoder_hidden_states": torch.randn(2, 77, 768),
- "down_block_additional_residuals": down_block_res_samples,
- "mid_block_additional_residual": mid_block_res_sample,
- }
- if not UNET_CONTROLNET_PATH.exists():
- with torch.no_grad():
- ov_unet = ov.convert_model(UnetWrapper(unet), example_input=example_input)
- flat_inputs = flatten_inputs(example_input.values())
- for input_data, input_tensor in zip(flat_inputs, ov_unet.inputs):
- input_tensor.get_node().set_partial_shape(ov.PartialShape(input_data.shape))
- input_tensor.get_node().set_element_type(dtype_mapping[input_data.dtype])
- ov_unet.validate_nodes_and_infer_types()
- ov.save_model(ov_unet, UNET_CONTROLNET_PATH)
- del ov_unet
- del unet
- gc.collect()
-
-
-
-
-.. parsed-literal::
-
- 0
-
-
-
-Variational Autoencoder (VAE)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-The variational autoencoder (VAE) model with KL loss was introduced in
-`Auto-Encoding Variational
-Bayes `__. The model is used to
-encode images into latents and to decode latent representations into
-images. For inference we use only decoding part of the VAE. We wrap the
-decoder in separate ``torch.nn.Module``.
-
-.. code:: ipython3
-
- class VaeDecoderWrapper(torch.nn.Module):
- def __init__(self, vae: torch.nn.Module):
- super().__init__()
- self.vae = vae
-
- def forward(self, z: torch.FloatTensor):
- return self.vae.decode(z / self.vae.config.scaling_factor, return_dict=False)[0]
-
-
- serialize_openvino(
- VaeDecoderWrapper(vae),
- VAE_PATH,
- example_input=torch.randn(1, 4, 64, 64),
- input=((1, 4, 64, 64)),
- )
- del vae
- gc.collect()
-
-
-
-
-.. parsed-literal::
-
- 0
-
-
-
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- core = ov.Core()
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=4, options=('CPU', 'GPU.0', 'GPU.1', 'GPU.2', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- qformer = core.compile_model(QFORMER_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- text_encoder = core.compile_model(TEXT_ENCODER_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- neg_text_encoder = core.compile_model(NEG_TEXT_ENCODER_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- controlnet = core.compile_model(CONTROLNET_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- unet = core.compile_model(UNET_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- unet_controlnet = core.compile_model(UNET_CONTROLNET_PATH, device_name=device.value)
-
-.. code:: ipython3
-
- vae = core.compile_model(VAE_PATH, device_name=device.value)
-
-Inference
----------
-
-
-
-.. code:: ipython3
-
- def call(compiled_model, *args, **kwargs):
- if len(args) and not kwargs:
- result = compiled_model([np.array(a) for a in args])[0]
- elif kwargs and not len(args):
- result = compiled_model({k: np.array(v) for k, v in kwargs.items()})[0]
- else:
- raise NotImplementedError(f"{args=},{kwargs=}")
- result = torch.tensor(result)
- return result
-
-.. code:: ipython3
-
- class OvBlipDiffusionPipeline(diffusers.DiffusionPipeline):
- def __init__(self):
- self.tokenizer = tokenizer
- self.qformer_tokenizer = qformer_tokenizer
- self.text_encoder = partial(call, text_encoder)
- self.neg_text_encoder = partial(call, neg_text_encoder)
- self.vae = partial(call, vae)
- self.unet = partial(call, unet)
- self.unet_controlnet = partial(call, unet_controlnet)
- self.controlnet = controlnet
- self.scheduler = scheduler
- self.qformer = partial(call, qformer)
- self.image_processor = image_processor
- self.register_to_config(**config)
-
- def __call__(
- self,
- prompt: List[str],
- reference_image: PIL.Image.Image,
- source_subject_category: List[str],
- target_subject_category: List[str],
- conditioning_image: Optional[PIL.Image.Image] = None,
- latents: Optional[torch.FloatTensor] = None,
- guidance_scale: float = 7.5,
- num_inference_steps: int = 50,
- generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
- neg_prompt: Optional[str] = "",
- prompt_strength: float = 1.0,
- prompt_reps: int = 20,
- output_type: Optional[str] = "pil",
- ):
- """
- Function invoked when calling the pipeline for generation.
-
- Args:
- prompt (`List[str]`):
- The prompt or prompts to guide the image generation.
- reference_image (`PIL.Image.Image`):
- The reference image to condition the generation on.
- source_subject_category (`List[str]`):
- The source subject category.
- target_subject_category (`List[str]`):
- The target subject category.
- conditioning_image (`PIL.Image.Image`):
- The conditioning canny edge image to condition the generation on.
- latents (`torch.FloatTensor`, *optional*):
- Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
- generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
- tensor will ge generated by random sampling.
- guidance_scale (`float`, *optional*, defaults to 7.5):
- Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
- `guidance_scale` is defined as `w` of equation 2. of [Imagen
- Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
- 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
- usually at the expense of lower image quality.
- num_inference_steps (`int`, *optional*, defaults to 50):
- The number of denoising steps. More denoising steps usually lead to a higher quality image at the
- expense of slower inference.
- generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
- One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
- to make generation deterministic.
- neg_prompt (`str`, *optional*, defaults to ""):
- The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored
- if `guidance_scale` is less than `1`).
- prompt_strength (`float`, *optional*, defaults to 1.0):
- The strength of the prompt. Specifies the number of times the prompt is repeated along with prompt_reps
- to amplify the prompt.
- prompt_reps (`int`, *optional*, defaults to 20):
- The number of times the prompt is repeated along with prompt_strength to amplify the prompt.
- output_type (`str`, *optional*, defaults to `"pil"`):
- The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"`
- (`np.array`) or `"pt"` (`torch.Tensor`).
- """
- width = 512
- height = 512
- reference_image = self.image_processor.preprocess(
- reference_image,
- image_mean=self.config.mean,
- image_std=self.config.std,
- return_tensors="pt",
- )["pixel_values"]
-
- if isinstance(prompt, str):
- prompt = [prompt]
- if isinstance(source_subject_category, str):
- source_subject_category = [source_subject_category]
- if isinstance(target_subject_category, str):
- target_subject_category = [target_subject_category]
-
- batch_size = len(prompt)
-
- prompt = self._build_prompt(
- prompts=prompt,
- tgt_subjects=target_subject_category,
- prompt_strength=prompt_strength,
- prompt_reps=prompt_reps,
- )
- qformer_input = self.qformer_tokenizer(source_subject_category, return_tensors="pt", padding=True)
- query_embeds = self.qformer(
- image_input=reference_image,
- text_input_ids=qformer_input.input_ids,
- text_attention_mask=qformer_input.attention_mask,
- )
- text_embeddings = self.encode_prompt(query_embeds, prompt, device)
- do_classifier_free_guidance = guidance_scale > 1.0
- if do_classifier_free_guidance:
- max_length = self.config.text_encoder_max_position_embeddings
-
- uncond_input = self.tokenizer(
- [neg_prompt] * batch_size,
- padding="max_length",
- max_length=max_length,
- return_tensors="pt",
- )
- uncond_embeddings = self.neg_text_encoder(input_ids=uncond_input.input_ids)
- # For classifier free guidance, we need to do two forward passes.
- # Here we concatenate the unconditional and text embeddings into a single batch
- # to avoid doing two forward passes
- text_embeddings = torch.cat([uncond_embeddings, text_embeddings])
-
- scale_down_factor = 2 ** (len(self.config.unet_block_out_channels) - 1)
- latents = self.prepare_latents(
- batch_size=batch_size,
- num_channels=self.config.unet_in_channels,
- height=height // scale_down_factor,
- width=width // scale_down_factor,
- generator=generator,
- latents=latents,
- device=None,
- dtype=None,
- )
- # set timesteps
- extra_set_kwargs = {}
- self.scheduler.set_timesteps(num_inference_steps, **extra_set_kwargs)
-
- if conditioning_image:
- cond_image = self.prepare_control_image(
- image=conditioning_image,
- width=width,
- height=height,
- batch_size=batch_size,
- num_images_per_prompt=1,
- device=None,
- dtype=None,
- do_classifier_free_guidance=do_classifier_free_guidance,
- )
- for i, t in enumerate(self.progress_bar(self.scheduler.timesteps)):
- # expand the latents if we are doing classifier free guidance
- do_classifier_free_guidance = guidance_scale > 1.0
-
- latent_model_input = torch.cat([latents] * 2) if do_classifier_free_guidance else latents
- if conditioning_image:
- controlnet_output = self.controlnet(
- [
- latent_model_input,
- t,
- text_embeddings,
- cond_image,
- ]
- )
- noise_pred = (
- self.unet(
- sample=latent_model_input,
- timestep=t,
- encoder_hidden_states=text_embeddings,
- )
- if not conditioning_image
- else self.unet_controlnet(
- latent_model_input,
- t,
- text_embeddings,
- *[v for _, v in controlnet_output.items()],
- )
- )
-
- # perform guidance
- if do_classifier_free_guidance:
- noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)
- noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)
-
- latents = self.scheduler.step(
- noise_pred,
- t,
- latents,
- )["prev_sample"]
-
- image = self.vae(latents)
- image = self.image_processor.postprocess(image, output_type=output_type)
- return image
-
- def encode_prompt(self, query_embeds, prompt, device=None):
- # embeddings for prompt, with query_embeds as context
- max_len = self.config.text_encoder_max_position_embeddings
- max_len -= self.config.qformer_num_query_tokens
-
- tokenized_prompt = self.tokenizer(
- prompt,
- padding="max_length",
- truncation=True,
- max_length=max_len,
- return_tensors="pt",
- )
-
- batch_size = query_embeds.shape[0]
- ctx_begin_pos = [self.config.ctx_begin_pos] * batch_size
-
- text_embeddings = self.text_encoder(
- input_ids=tokenized_prompt.input_ids,
- ctx_embeddings=query_embeds,
- ctx_begin_pos=ctx_begin_pos,
- )
-
- return text_embeddings
-
-
- OvBlipDiffusionPipeline.prepare_control_image = diffusers.pipelines.BlipDiffusionControlNetPipeline.prepare_control_image
- OvBlipDiffusionPipeline._build_prompt = diffusers.pipelines.BlipDiffusionPipeline._build_prompt
- OvBlipDiffusionPipeline.prepare_latents = diffusers.pipelines.BlipDiffusionPipeline.prepare_latents
-
-.. code:: ipython3
-
- ov_pipe = OvBlipDiffusionPipeline()
-
-Zero-Shot subject-driven generation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- output = ov_pipe(
- text_prompt_input,
- dog_img,
- cond_subject,
- tgt_subject,
- guidance_scale=guidance_scale,
- num_inference_steps=num_inference_steps,
- neg_prompt=negative_prompt,
- )
-
-
-
-.. parsed-literal::
-
- 0%| | 0/51 [00:00`__
-on a pre-trained model using Hugging Face Optimum and benchmark
-performance.
-
-Feel free to use the notebook outline in Jupyter or your IDE for easy
-navigation.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Introduction <#introduction>`__
-
- - `Task <#task>`__
- - `Model <#model>`__
-
-- `Part 1: Inference pipeline with
- OpenVINO <#part-1-inference-pipeline-with-openvino>`__
-
- - `Install prerequisites <#install-prerequisites>`__
- - `Imports <#imports>`__
- - `Setting up HuggingFace cache <#setting-up-huggingface-cache>`__
- - `Select inference device <#select-inference-device>`__
- - `Download resources <#download-resources>`__
- - `Create inference pipeline <#create-inference-pipeline>`__
- - `Inference on new input <#inference-on-new-input>`__
-
-- `Part 2: OpenVINO post-training quantization with HuggingFace
- Optimum <#part-2-openvino-post-training-quantization-with-huggingface-optimum>`__
-
- - `Define constants and
- functions <#define-constants-and-functions>`__
- - `Load resources <#load-resources>`__
- - `Load calibration dataset <#load-calibration-dataset>`__
- - `Quantize model <#quantize-model>`__
- - `Load quantized model <#load-quantized-model>`__
- - `Inference on new input using quantized
- model <#inference-on-new-input-using-quantized-model>`__
- - `Load evaluation set <#load-evaluation-set>`__
- - `Evaluate model <#evaluate-model>`__
-
-- `Additional resources <#additional-resources>`__
-- `Clean up <#clean-up>`__
-
-Introduction
-------------
-
-
-
-Task
-~~~~
-
-
-
-**Programming language classification** is the task of identifying which
-programming language is used in an arbitrary code snippet. This can be
-useful to label new data to include in a dataset, and potentially serve
-as an intermediary step when input snippets need to be process based on
-their programming language.
-
-It is a relatively easy machine learning task given that each
-programming language has its own formal symbols, syntax, and grammar.
-However, there are some potential edge cases: - **Ambiguous short
-snippets**: For example, TypeScript is a superset of JavaScript, meaning
-it does everything JavaScript can and more. For a short input snippet,
-it might be impossible to distinguish between the two. Given we know
-TypeScript is a superset, and the model doesn’t, we should default to
-classifying the input as JavaScript in a post-processing step. -
-**Nested programming languages**: Some languages are typically used in
-tandem. For example, most HTML contains CSS and JavaScript, and it is
-not uncommon to see SQL nested in other scripting languages. For such
-input, it is unclear what the expected output class should be. -
-**Evolving programming language**: Even though programming languages are
-formal, their symbols, syntax, and grammar can be revised and updated.
-For example, the walrus operator (``:=``) was a symbol distinctively
-used in Golang, but was later introduced in Python 3.8.
-
-Model
-~~~~~
-
-
-
-The classification model that will be used in this notebook is
-`CodeBERTa-language-id `__
-by HuggingFace. This model was fine-tuned from the masked language
-modeling model
-`CodeBERTa-small-v1 `__
-trained on the
-`CodeSearchNet `__
-dataset (Husain, 2019).
-
-It supports 6 programming languages: - Go - Java - JavaScript - PHP -
-Python - Ruby
-
-Part 1: Inference pipeline with OpenVINO
-----------------------------------------
-
-
-
-For this section, we will use the `HuggingFace
-Optimum `__ library, which
-aims to optimize inference on specific hardware and integrates with the
-OpenVINO toolkit. The code will be very similar to the `HuggingFace
-Transformers `__, but
-will allow to automatically convert models to the OpenVINO™ IR format.
-
-Install prerequisites
-~~~~~~~~~~~~~~~~~~~~~
-
-
-
-First, complete the `repository installation steps <../../README.md>`__.
-
-Then, the following cell will install: - HuggingFace Optimum with
-OpenVINO support - HuggingFace Evaluate to benchmark results
-
-.. code:: ipython3
-
- %pip install -q "diffusers>=0.17.1" "openvino>=2023.1.0" "nncf>=2.5.0" "gradio>=4.19" "onnx>=1.11.0" "transformers>=4.33.0" "torch>=2.1" "evaluate" --extra-index-url https://download.pytorch.org/whl/cpu
- %pip install -q "git+https://github.com/huggingface/optimum-intel.git"
-
-
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-Imports
-~~~~~~~
-
-
-
-The import ``OVModelForSequenceClassification`` from Optimum is
-equivalent to ``AutoModelForSequenceClassification`` from Transformers
-
-.. code:: ipython3
-
- from functools import partial
- from pathlib import Path
-
- import pandas as pd
- from datasets import load_dataset, Dataset
- import evaluate
- from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
- from optimum.intel import OVModelForSequenceClassification
- from optimum.intel.openvino import OVConfig, OVQuantizer, OVWeightQuantizationConfig
- from huggingface_hub.utils import RepositoryNotFoundError
-
-
-.. parsed-literal::
-
- 2024-04-17 23:25:25.155476: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:25:25.190609: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
- To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:25:25.785655: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
- torch.utils._pytree._register_pytree_node(
-
-
-Setting up HuggingFace cache
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Resources from HuggingFace will be downloaded in the local folder
-``./model`` (next to this notebook) instead of the device global cache
-for easy cleanup. Learn more
-`here `__.
-
-.. code:: ipython3
-
- MODEL_NAME = "CodeBERTa-language-id"
- MODEL_ID = f"huggingface/{MODEL_NAME}"
- MODEL_LOCAL_PATH = Path("./model").joinpath(MODEL_NAME)
-
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
- import openvino as ov
-
- core = ov.Core()
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-Download resources
-~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- # try to load resources locally
- try:
- model = OVModelForSequenceClassification.from_pretrained(MODEL_LOCAL_PATH, device=device.value)
- tokenizer = AutoTokenizer.from_pretrained(MODEL_LOCAL_PATH)
- print(f"Loaded resources from local path: {MODEL_LOCAL_PATH.absolute()}")
-
- # if not found, download from HuggingFace Hub then save locally
- except (RepositoryNotFoundError, OSError):
- print("Downloading resources from HuggingFace Hub")
- tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
- tokenizer.save_pretrained(MODEL_LOCAL_PATH)
-
- # export=True is needed to convert the PyTorch model to OpenVINO
- model = OVModelForSequenceClassification.from_pretrained(MODEL_ID, export=True, device=device.value)
- model.save_pretrained(MODEL_LOCAL_PATH)
- print(f"Ressources cached locally at: {MODEL_LOCAL_PATH.absolute()}")
-
-
-.. parsed-literal::
-
- Downloading resources from HuggingFace Hub
-
-
-.. parsed-literal::
-
- Framework not specified. Using pt to export the model.
-
-
-.. parsed-literal::
-
- Some weights of the model checkpoint at huggingface/CodeBERTa-language-id were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- - This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- - This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
-
-
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
-
-
-.. parsed-literal::
-
- Overriding 1 configuration item(s)
-
-
-.. parsed-literal::
-
- - use_cache -> False
-
-
-.. parsed-literal::
-
- WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
-
-
-.. parsed-literal::
-
- [ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
- warnings.warn(
-
-
-.. parsed-literal::
-
- Compiling the model to AUTO ...
-
-
-.. parsed-literal::
-
- Ressources cached locally at: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/code-language-id/model/CodeBERTa-language-id
-
-
-Create inference pipeline
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- code_classification_pipe = pipeline("text-classification", model=model, tokenizer=tokenizer)
-
-Inference on new input
-~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- # change input snippet to test model
- input_snippet = "df['speed'] = df.distance / df.time"
- output = code_classification_pipe(input_snippet)
-
- print(f"Input snippet:\n {input_snippet}\n")
- print(f"Predicted label: {output[0]['label']}")
- print(f"Predicted score: {output[0]['score']:.2}")
-
-
-.. parsed-literal::
-
- Input snippet:
- df['speed'] = df.distance / df.time
-
- Predicted label: python
- Predicted score: 0.81
-
-
-Part 2: OpenVINO post-training quantization with HuggingFace Optimum
---------------------------------------------------------------------
-
-
-
-In this section, we will quantize a trained model. At a high-level, this
-process consists of using lower precision numbers in the model, which
-results in a smaller model size and faster inference at the cost of a
-potential marginal performance degradation. `Learn
-more `__.
-
-The HuggingFace Optimum library supports post-training quantization for
-OpenVINO. `Learn
-more `__.
-
-Define constants and functions
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- QUANTIZED_MODEL_LOCAL_PATH = MODEL_LOCAL_PATH.with_name(f"{MODEL_NAME}-quantized")
- DATASET_NAME = "code_search_net"
- LABEL_MAPPING = {"go": 0, "java": 1, "javascript": 2, "php": 3, "python": 4, "ruby": 5}
-
-
- def preprocess_function(examples: dict, tokenizer):
- """Preprocess inputs by tokenizing the `func_code_string` column"""
- return tokenizer(
- examples["func_code_string"],
- padding="max_length",
- max_length=tokenizer.model_max_length,
- truncation=True,
- )
-
-
- def map_labels(example: dict) -> dict:
- """Convert string labels to integers"""
- label_mapping = {
- "go": 0,
- "java": 1,
- "javascript": 2,
- "php": 3,
- "python": 4,
- "ruby": 5,
- }
- example["language"] = label_mapping[example["language"]]
- return example
-
-
- def get_dataset_sample(dataset_split: str, num_samples: int) -> Dataset:
- """Create a sample with equal representation of each class without downloading the entire data"""
- labels = ["go", "java", "javascript", "php", "python", "ruby"]
- example_per_label = num_samples // len(labels)
-
- examples = []
- for label in labels:
- subset = load_dataset("code_search_net", split=dataset_split, name=label, streaming=True)
- subset = subset.map(map_labels)
- examples.extend([example for example in subset.shuffle().take(example_per_label)])
-
- return Dataset.from_list(examples)
-
-Load resources
-~~~~~~~~~~~~~~
-
-
-
-NOTE: the base model is loaded using
-``AutoModelForSequenceClassification`` from ``Transformers``
-
-.. code:: ipython3
-
- tokenizer = AutoTokenizer.from_pretrained(MODEL_LOCAL_PATH)
- base_model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
-
- quantizer = OVQuantizer.from_pretrained(base_model)
- quantization_config = OVWeightQuantizationConfig()
- ov_config = OVConfig(quantization_config=quantization_config)
-
-
-.. parsed-literal::
-
- Some weights of the model checkpoint at huggingface/CodeBERTa-language-id were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- - This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- - This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
-
-
-Load calibration dataset
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-The ``get_dataset_sample()`` function will sample up to ``num_samples``,
-with an equal number of examples across the 6 programming languages.
-
-NOTE: Uncomment the method below to download and use the full dataset
-(5+ Gb).
-
-.. code:: ipython3
-
- calibration_sample = get_dataset_sample(dataset_split="train", num_samples=120)
- calibration_sample = calibration_sample.map(partial(preprocess_function, tokenizer=tokenizer))
-
- # calibration_sample = quantizer.get_calibration_dataset(
- # DATASET_NAME,
- # preprocess_function=partial(preprocess_function, tokenizer=tokenizer),
- # num_samples=120,
- # dataset_split="train",
- # preprocess_batch=True,
- # )
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1461: FutureWarning: The repository for code_search_net contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/code_search_net
- You can avoid this message in future by passing the argument `trust_remote_code=True`.
- Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
- warnings.warn(
-
-
-
-.. parsed-literal::
-
- Map: 0%| | 0/120 [00:00
-
-
-
-
-.. raw:: html
-
-
-
-
-
-
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
-
-
-.. parsed-literal::
-
- Overriding 1 configuration item(s)
-
-
-.. parsed-literal::
-
- - use_cache -> False
-
-
-.. parsed-literal::
-
- WARNING:nncf:You are setting `forward` on an NNCF-processed model object.
- NNCF relies on custom-wrapping the `forward` call in order to function properly.
- Arbitrary adjustments to the forward function on an NNCFNetwork object have undefined behavior.
- If you need to replace the underlying forward function of the original model so that NNCF should be using that instead of the original forward function that NNCF saved during the compressed model creation, you can do this by calling:
- model.nncf.set_original_unbound_forward(fn)
- if `fn` has an unbound 0-th `self` argument, or
- with model.nncf.temporary_bound_original_forward(fn): ...
- if `fn` already had 0-th `self` argument bound or never had it in the first place.
-
-
-.. parsed-literal::
-
- WARNING:nncf:You are setting `forward` on an NNCF-processed model object.
- NNCF relies on custom-wrapping the `forward` call in order to function properly.
- Arbitrary adjustments to the forward function on an NNCFNetwork object have undefined behavior.
- If you need to replace the underlying forward function of the original model so that NNCF should be using that instead of the original forward function that NNCF saved during the compressed model creation, you can do this by calling:
- model.nncf.set_original_unbound_forward(fn)
- if `fn` has an unbound 0-th `self` argument, or
- with model.nncf.temporary_bound_original_forward(fn): ...
- if `fn` already had 0-th `self` argument bound or never had it in the first place.
-
-
-.. parsed-literal::
-
- Configuration saved in model/CodeBERTa-language-id-quantized/openvino_config.json
-
-
-Load quantized model
-~~~~~~~~~~~~~~~~~~~~
-
-
-
-NOTE: the argument ``export=True`` is not required since the quantized
-model is already in the OpenVINO format.
-
-.. code:: ipython3
-
- quantized_model = OVModelForSequenceClassification.from_pretrained(QUANTIZED_MODEL_LOCAL_PATH, device=device.value)
- quantized_code_classification_pipe = pipeline("text-classification", model=quantized_model, tokenizer=tokenizer)
-
-
-.. parsed-literal::
-
- Compiling the model to AUTO ...
-
-
-Inference on new input using quantized model
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- input_snippet = "df['speed'] = df.distance / df.time"
- output = quantized_code_classification_pipe(input_snippet)
-
- print(f"Input snippet:\n {input_snippet}\n")
- print(f"Predicted label: {output[0]['label']}")
- print(f"Predicted score: {output[0]['score']:.2}")
-
-
-.. parsed-literal::
-
- Input snippet:
- df['speed'] = df.distance / df.time
-
- Predicted label: python
- Predicted score: 0.81
-
-
-Load evaluation set
-~~~~~~~~~~~~~~~~~~~
-
-
-
-NOTE: Uncomment the method below to download and use the full dataset
-(5+ Gb).
-
-.. code:: ipython3
-
- validation_sample = get_dataset_sample(dataset_split="validation", num_samples=120)
-
- # validation_sample = load_dataset(DATASET_NAME, split="validation")
-
-Evaluate model
-~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- # This class is needed due to a current limitation of the Evaluate library with multiclass metrics
- # ref: https://discuss.huggingface.co/t/combining-metrics-for-multiclass-predictions-evaluations/21792/16
- class ConfiguredMetric:
- def __init__(self, metric, *metric_args, **metric_kwargs):
- self.metric = metric
- self.metric_args = metric_args
- self.metric_kwargs = metric_kwargs
-
- def add(self, *args, **kwargs):
- return self.metric.add(*args, **kwargs)
-
- def add_batch(self, *args, **kwargs):
- return self.metric.add_batch(*args, **kwargs)
-
- def compute(self, *args, **kwargs):
- return self.metric.compute(*args, *self.metric_args, **kwargs, **self.metric_kwargs)
-
- @property
- def name(self):
- return self.metric.name
-
- def _feature_names(self):
- return self.metric._feature_names()
-
-First, an ``Evaluator`` object for ``text-classification`` and a set of
-``EvaluationModule`` are instantiated. Then, the evaluator
-``.compute()`` method is called on both the base
-``code_classification_pipe`` and the quantized
-``quantized_code_classification_pipeline``. Finally, results are
-displayed.
-
-.. code:: ipython3
-
- code_classification_evaluator = evaluate.evaluator("text-classification")
- # instantiate an object that can contain multiple `evaluate` metrics
- metrics = evaluate.combine(
- [
- ConfiguredMetric(evaluate.load("f1"), average="macro"),
- ]
- )
-
- base_results = code_classification_evaluator.compute(
- model_or_pipeline=code_classification_pipe,
- data=validation_sample,
- input_column="func_code_string",
- label_column="language",
- label_mapping=LABEL_MAPPING,
- metric=metrics,
- )
-
- quantized_results = code_classification_evaluator.compute(
- model_or_pipeline=quantized_code_classification_pipe,
- data=validation_sample,
- input_column="func_code_string",
- label_column="language",
- label_mapping=LABEL_MAPPING,
- metric=metrics,
- )
-
- results_df = pd.DataFrame.from_records([base_results, quantized_results], index=["base", "quantized"])
- results_df
-
-
-
-
-.. raw:: html
-
-
-
-
-
-
-
-
f1
-
total_time_in_seconds
-
samples_per_second
-
latency_in_seconds
-
-
-
-
-
base
-
1.0
-
2.077464
-
57.762723
-
0.017312
-
-
-
quantized
-
1.0
-
1.988597
-
60.344039
-
0.016572
-
-
-
-
-
-
-
-Additional resources
---------------------
-
- - `Grammatical Error Correction
-with OpenVINO `__ -
-`Quantize a Hugging Face Question-Answering Model with
-OpenVINO `__\ \*\*
-
-Clean up
---------
-
-
-
-Uncomment and run cell below to delete all resources cached locally in
-./model
-
-.. code:: ipython3
-
- # import os
- # import shutil
-
- # try:
- # shutil.rmtree(path=QUANTIZED_MODEL_LOCAL_PATH)
- # shutil.rmtree(path=MODEL_LOCAL_PATH)
- # os.remove(path="./compressed_graph.dot")
- # os.remove(path="./original_graph.dot")
- # except FileNotFoundError:
- # print("Directory was already deleted")
diff --git a/docs/notebooks/convert-to-openvino-with-output.rst b/docs/notebooks/convert-to-openvino-with-output.rst
index fcc99b2e27d..4e38d7b6c19 100644
--- a/docs/notebooks/convert-to-openvino-with-output.rst
+++ b/docs/notebooks/convert-to-openvino-with-output.rst
@@ -35,16 +35,8 @@ Table of contents:
.. parsed-literal::
- Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
-
-
-.. parsed-literal::
-
+ Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -189,19 +181,13 @@ NLP model from Hugging Face and export it in ONNX format:
.. parsed-literal::
- 2024-04-17 23:27:23.489620: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:27:23.524697: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-06 23:46:51.110172: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-06 23:46:51.145296: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:27:24.035165: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
+ 2024-05-06 23:46:51.660347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)
@@ -474,10 +460,6 @@ To convert a model to OpenVINO IR, use the following API:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
-
-
-.. parsed-literal::
-
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@@ -521,10 +503,6 @@ documentation.
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
-
-
-.. parsed-literal::
-
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@@ -558,10 +536,6 @@ conversion API parameter as ``-1`` or ``?`` when using ``ovc``:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
-
-
-.. parsed-literal::
-
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@@ -606,10 +580,6 @@ sequence length dimension:
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression by removing argument "compress_to_fp16" or set it to false "compress_to_fp16=False".
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
-
-
-.. parsed-literal::
-
[ SUCCESS ] XML file: model/distilbert.xml
[ SUCCESS ] BIN file: model/distilbert.bin
@@ -694,12 +664,12 @@ frameworks conversion guides.
.. parsed-literal::
- 2024-04-17 23:27:44.237192: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
- 2024-04-17 23:27:44.237227: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
- 2024-04-17 23:27:44.237231: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
- 2024-04-17 23:27:44.237454: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
- 2024-04-17 23:27:44.237472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
- 2024-04-17 23:27:44.237475: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
+ 2024-05-06 23:47:11.917183: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
+ 2024-05-06 23:47:11.917219: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: iotg-dev-workstation-07
+ 2024-05-06 23:47:11.917224: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: iotg-dev-workstation-07
+ 2024-05-06 23:47:11.917431: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 470.223.2
+ 2024-05-06 23:47:11.917454: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 470.182.3
+ 2024-05-06 23:47:11.917459: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:312] kernel version 470.182.3 does not match DSO version 470.223.2 -- cannot find working devices in this configuration
Migration from Legacy conversion API
diff --git a/docs/notebooks/convnext-classification-with-output.rst b/docs/notebooks/convnext-classification-with-output.rst
index b41fde8985c..b6b2d1166a1 100644
--- a/docs/notebooks/convnext-classification-with-output.rst
+++ b/docs/notebooks/convnext-classification-with-output.rst
@@ -51,20 +51,8 @@ Prerequisites
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -195,7 +183,7 @@ And print results
Predicted Class: 281
Predicted Label: n02123045 tabby, tabby cat
- Predicted Probability: 0.5351971983909607
+ Predicted Probability: 0.5510364174842834
Convert the model to OpenVINO Intermediate representation format
diff --git a/docs/notebooks/cross-lingual-books-alignment-with-output.rst b/docs/notebooks/cross-lingual-books-alignment-with-output.rst
index 37a97fb5af4..2df720172b1 100644
--- a/docs/notebooks/cross-lingual-books-alignment-with-output.rst
+++ b/docs/notebooks/cross-lingual-books-alignment-with-output.rst
@@ -407,12 +407,12 @@ languages. It has the same architecture as the BERT model but has been
trained on a different task: to produce identical embeddings for
translation pairs.
-|image01|
+|image1|
This makes LaBSE a great choice for our task and it can be reused for
different language pairs still producing good results.
-.. |image01| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/627d3a39-7076-479f-a7b1-392f49a0b83e
+.. |image1| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/627d3a39-7076-479f-a7b1-392f49a0b83e
.. code:: ipython3
diff --git a/docs/notebooks/ct-scan-live-inference-with-output.rst b/docs/notebooks/ct-scan-live-inference-with-output.rst
deleted file mode 100644
index 286355536e2..00000000000
--- a/docs/notebooks/ct-scan-live-inference-with-output.rst
+++ /dev/null
@@ -1,515 +0,0 @@
-Live Inference and Benchmark CT-scan Data with OpenVINO™
-========================================================
-
-Kidney Segmentation with PyTorch Lightning and OpenVINO™ - Part 4
------------------------------------------------------------------
-
-This tutorial is a part of a series on how to train, optimize, quantize
-and show live inference on a medical segmentation model. The goal is to
-accelerate inference on a kidney segmentation model. The
-`UNet `__ model is trained from
-scratch, and the data is from
-`Kits19 `__.
-
-This tutorial shows how to benchmark performance of the model and show
-live inference with async API and MULTI plugin in OpenVINO.
-
-This notebook needs a quantized OpenVINO IR model and images from the
-`KiTS-19 `__ dataset, converted to
-2D images. (To learn how the model is quantized, see the `Convert and
-Quantize a UNet Model and Show Live
-Inference `__ tutorial.)
-
-This notebook provides a pre-trained model, trained for 20 epochs with
-the full KiTS-19 frames dataset, which has an F1 score on the validation
-set of 0.9. The training code is available in the `PyTorch MONAI
-Training `__
-notebook.
-
-For demonstration purposes, this tutorial will download one converted CT
-scan to use for inference.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Imports <#imports>`__
-- `Settings <#settings>`__
-- `Benchmark Model Performance <#benchmark-model-performance>`__
-- `Download and Prepare Data <#download-and-prepare-data>`__
-- `Show Live Inference <#show-live-inference>`__
-
- - `Load Model and List of Image
- Files <#load-model-and-list-of-image-files>`__
- - `Prepare images <#prepare-images>`__
- - `Specify device <#specify-device>`__
- - `Setting callback function <#setting-callback-function>`__
- - `Create asynchronous inference queue and perform
- it <#create-asynchronous-inference-queue-and-perform-it>`__
-
-.. code:: ipython3
-
- %pip install -q "openvino>=2023.3.0" "monai>=0.9.1" "nncf>=2.8.0" "opencv-python" "tqdm"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-Imports
--------
-
-
-
-.. code:: ipython3
-
- import os
- import zipfile
- from pathlib import Path
-
- import numpy as np
- from monai.transforms import LoadImage
- import openvino as ov
-
- from custom_segmentation import SegmentationModel
-
- # Fetch `notebook_utils` module
- import requests
-
- r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py")
- open("notebook_utils.py", "w").write(r.text)
- from notebook_utils import download_file
-
-
-.. parsed-literal::
-
- 2024-04-17 23:29:02.693500: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:29:02.729260: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
- To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:29:03.313117: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-Settings
---------
-
-
-
-To use the pre-trained models, set ``IR_PATH`` to
-``"pretrained_model/unet44.xml"`` and ``COMPRESSED_MODEL_PATH`` to
-``"pretrained_model/quantized_unet44.xml"``. To use a model that you
-trained or optimized yourself, adjust the model paths.
-
-.. code:: ipython3
-
- # The directory that contains the IR model (xml and bin) files.
- models_dir = Path("pretrained_model")
-
- ir_model_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/FP16-INT8/"
- ir_model_name_xml = "quantized_unet_kits19.xml"
- ir_model_name_bin = "quantized_unet_kits19.bin"
-
- download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory=models_dir)
- download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory=models_dir)
-
- MODEL_PATH = models_dir / ir_model_name_xml
-
- # Uncomment the next line to use the FP16 model instead of the quantized model.
- # MODEL_PATH = "pretrained_model/unet_kits19.xml"
-
-
-
-.. parsed-literal::
-
- pretrained_model/quantized_unet_kits19.xml: 0%| | 0.00/280k [00:00`__
-- an inference performance measurement tool in OpenVINO. Benchmark tool
-is a command-line application that can be run in the notebook with
-``! benchmark_app`` or ``%sx benchmark_app`` commands.
-
- **Note**: The ``benchmark_app`` tool is able to measure the
- performance of the OpenVINO Intermediate Representation (OpenVINO IR)
- models only. For more accurate performance, run ``benchmark_app`` in
- a terminal/command prompt after closing other applications. Run
- ``benchmark_app -m model.xml -d CPU`` to benchmark async inference on
- CPU for one minute. Change ``CPU`` to ``GPU`` to benchmark on GPU.
- Run ``benchmark_app --help`` to see an overview of all command-line
- options.
-
-.. code:: ipython3
-
- core = ov.Core()
- # By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.
- device_list = ["MULTI:CPU,GPU" if "GPU" in core.available_devices else "AUTO"]
-
- import ipywidgets as widgets
-
- device = widgets.Dropdown(
- options=core.available_devices + device_list,
- value=device_list[0],
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- # Benchmark model
- ! benchmark_app -m $MODEL_PATH -d $device.value -t 15 -api sync
-
-
-.. parsed-literal::
-
- [Step 1/11] Parsing and validating input arguments
- [ INFO ] Parsing input parameters
- [Step 2/11] Loading OpenVINO Runtime
- [ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ] Device info:
- [ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ]
- [Step 3/11] Setting device configuration
- [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
- [Step 4/11] Reading model files
- [ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 13.73 ms
- [ INFO ] Original model I/O parameters:
- [ INFO ] Model inputs:
- [ INFO ] input.1 (node: input.1) : f32 / [...] / [1,1,512,512]
- [ INFO ] Model outputs:
- [ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
- [Step 5/11] Resizing model to match image sizes and given batch
- [ INFO ] Model batch size: 1
- [Step 6/11] Configuring input of the model
- [ INFO ] Model inputs:
- [ INFO ] input.1 (node: input.1) : f32 / [N,C,H,W] / [1,1,512,512]
- [ INFO ] Model outputs:
- [ INFO ] 153 (node: 153) : f32 / [...] / [1,1,512,512]
- [Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 303.13 ms
- [Step 8/11] Querying optimal runtime parameters
- [ INFO ] Model:
- [ INFO ] NETWORK_NAME: pretrained_unet_kits19
- [ INFO ] EXECUTION_DEVICES: ['CPU']
- [ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
- [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
- [ INFO ] MULTI_DEVICE_PRIORITIES: CPU
- [ INFO ] CPU:
- [ INFO ] AFFINITY: Affinity.CORE
- [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
- [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
- [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
- [ INFO ] ENABLE_CPU_PINNING: True
- [ INFO ] ENABLE_HYPER_THREADING: False
- [ INFO ] EXECUTION_DEVICES: ['CPU']
- [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
- [ INFO ] INFERENCE_NUM_THREADS: 12
- [ INFO ] INFERENCE_PRECISION_HINT:
- [ INFO ] KV_CACHE_PRECISION:
- [ INFO ] LOG_LEVEL: Level.NO
- [ INFO ] NETWORK_NAME: pretrained_unet_kits19
- [ INFO ] NUM_STREAMS: 1
- [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
- [ INFO ] PERFORMANCE_HINT: LATENCY
- [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
- [ INFO ] PERF_COUNT: NO
- [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
- [ INFO ] MODEL_PRIORITY: Priority.MEDIUM
- [ INFO ] LOADED_FROM_CACHE: False
- [Step 9/11] Creating infer requests and preparing input tensors
- [ WARNING ] No input files were given for input 'input.1'!. This input will be filled with random values!
- [ INFO ] Fill input 'input.1' with random values
- [Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
- [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
- [ INFO ] First inference took 25.71 ms
-
-
-.. parsed-literal::
-
- [Step 11/11] Dumping statistics report
- [ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 1348 iterations
- [ INFO ] Duration: 15011.04 ms
- [ INFO ] Latency:
- [ INFO ] Median: 10.88 ms
- [ INFO ] Average: 10.94 ms
- [ INFO ] Min: 10.68 ms
- [ INFO ] Max: 13.65 ms
- [ INFO ] Throughput: 89.80 FPS
-
-
-Download and Prepare Data
--------------------------
-
-
-
-Download one validation video for live inference.
-
-This tutorial reuses the ``KitsDataset`` class that was also used in the
-training and quantization notebook that will be released later.
-
-The data is expected in ``BASEDIR``. The ``BASEDIR`` directory should
-contain the ``case_00000`` to ``case_00299`` subdirectories. If the data
-for the case specified above does not already exist, it will be
-downloaded and extracted in the next cell.
-
-.. code:: ipython3
-
- # Directory that contains the CT scan data. This directory should contain subdirectories
- # case_00XXX where XXX is between 000 and 299.
- BASEDIR = Path("kits19_frames_1")
- # The CT scan case number. For example: 16 for data from the case_00016 directory.
- # Currently only 117 is supported.
- CASE = 117
-
- case_path = BASEDIR / f"case_{CASE:05d}"
-
- if not case_path.exists():
- filename = download_file(f"https://storage.openvinotoolkit.org/data/test_data/openvino_notebooks/kits19/case_{CASE:05d}.zip")
- with zipfile.ZipFile(filename, "r") as zip_ref:
- zip_ref.extractall(path=BASEDIR)
- os.remove(filename) # remove zipfile
- print(f"Downloaded and extracted data for case_{CASE:05d}")
- else:
- print(f"Data for case_{CASE:05d} exists")
-
-
-
-.. parsed-literal::
-
- case_00117.zip: 0%| | 0.00/5.48M [00:00`__.
-
-We will use
-`AsyncInferQueue `__
-to perform asynchronous inference. It can be instantiated with compiled
-model and a number of jobs - parallel execution threads. If you don’t
-pass a number of jobs or pass ``0``, then OpenVINO will pick the optimal
-number based on your device and heuristics. After acquiring the
-inference queue, there are two jobs to do:
-
-- Preprocess the data and push it to the inference queue. The
- preprocessing steps will remain the same.
-- Tell the inference queue what to do with the model output after the
- inference is finished. It is represented by the ``callback`` python
- function that takes an inference result and data that we passed to
- the inference queue along with the prepared input data
-
-Everything else will be handled by the ``AsyncInferQueue`` instance.
-
-Load Model and List of Image Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Load the segmentation model to OpenVINO Runtime with
-``SegmentationModel``, based on the Model API from `Open Model
-Zoo `__. This model
-implementation includes pre and post processing for the model. For
-``SegmentationModel`` this includes the code to create an overlay of the
-segmentation mask on the original image/frame. Uncomment the next cell
-to see the implementation.
-
-.. code:: ipython3
-
- core = ov.Core()
- segmentation_model = SegmentationModel(ie=core, model_path=Path(MODEL_PATH), sigmoid=True, rotate_and_flip=True)
- image_paths = sorted(case_path.glob("imaging_frames/*jpg"))
-
- print(f"{case_path.name}, {len(image_paths)} images")
-
-
-.. parsed-literal::
-
- case_00117, 69 images
-
-
-Prepare images
-~~~~~~~~~~~~~~
-
-
-
-Use the ``reader = LoadImage()`` function to read the images in the same
-way as in the
-`training `__
-tutorial.
-
-.. code:: ipython3
-
- framebuf = []
-
- next_frame_id = 0
- reader = LoadImage(image_only=True, dtype=np.uint8)
-
- while next_frame_id < len(image_paths) - 1:
- image_path = image_paths[next_frame_id]
- image = reader(str(image_path))
- framebuf.append(image)
- next_frame_id += 1
-
-Specify device
-~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-Setting callback function
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-When ``callback`` is set, any job that ends the inference, calls the
-Python function. The ``callback`` function must have two arguments: one
-is the request that calls the ``callback``, which provides the
-``InferRequest`` API; the other is called ``userdata``, which provides
-the possibility of passing runtime values.
-
-The ``callback`` function will show the results of inference.
-
-.. code:: ipython3
-
- import cv2
- import copy
- from IPython import display
-
- from typing import Dict, Any
-
-
- # Define a callback function that runs every time the asynchronous pipeline completes inference on a frame
- def completion_callback(
- infer_request: ov.InferRequest,
- user_data: Dict[str, Any],
- ) -> None:
- preprocess_meta = user_data["preprocess_meta"]
-
- raw_outputs = {idx: copy.deepcopy(res.data) for idx, (out, res) in enumerate(zip(infer_request.model_outputs, infer_request.output_tensors))}
- frame = segmentation_model.postprocess(raw_outputs, preprocess_meta)
-
- _, encoded_img = cv2.imencode(".jpg", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])
- # Create IPython image
- i = display.Image(data=encoded_img)
-
- # Display the image in this notebook
- display.clear_output(wait=True)
- display.display(i)
-
-Create asynchronous inference queue and perform it
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- import time
-
- load_start_time = time.perf_counter()
- compiled_model = core.compile_model(segmentation_model.net, device.value)
- # Create asynchronous inference queue with optimal number of infer requests
- infer_queue = ov.AsyncInferQueue(compiled_model)
- infer_queue.set_callback(completion_callback)
- load_end_time = time.perf_counter()
-
- results = [None] * len(framebuf)
- frame_number = 0
-
- # Perform inference on every frame in the framebuffer
- start_time = time.time()
- for i, input_frame in enumerate(framebuf):
- inputs, preprocessing_meta = segmentation_model.preprocess({segmentation_model.net.input(0): input_frame})
- infer_queue.start_async(inputs, {"preprocess_meta": preprocessing_meta})
-
- # Wait until all inference requests in the AsyncInferQueue are completed
- infer_queue.wait_all()
- stop_time = time.time()
-
- # Calculate total inference time and FPS
- total_time = stop_time - start_time
- fps = len(framebuf) / total_time
- time_per_frame = 1 / fps
-
- print(f"Loaded model to {device} in {load_end_time-load_start_time:.2f} seconds.")
-
- print(f"Total time to infer all frames: {total_time:.3f}s")
- print(f"Time per frame: {time_per_frame:.6f}s ({fps:.3f} FPS)")
-
-
-
-.. image:: ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png
-
-
-.. parsed-literal::
-
- Loaded model to Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO') in 0.26 seconds.
- Total time to infer all frames: 2.480s
- Time per frame: 0.036477s (27.415 FPS)
-
diff --git a/docs/notebooks/ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png b/docs/notebooks/ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png
deleted file mode 100644
index 726f4f10b93..00000000000
--- a/docs/notebooks/ct-scan-live-inference-with-output_files/ct-scan-live-inference-with-output_21_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c11bf88f1b96b85ae9d8f26f04113364bdde7d822323cee25f6cb5b01bf4d93a
-size 48780
diff --git a/docs/notebooks/ct-segmentation-quantize-nncf-with-output.rst b/docs/notebooks/ct-segmentation-quantize-nncf-with-output.rst
index 1da3c580e02..99629c15f18 100644
--- a/docs/notebooks/ct-segmentation-quantize-nncf-with-output.rst
+++ b/docs/notebooks/ct-segmentation-quantize-nncf-with-output.rst
@@ -39,7 +39,7 @@ This notebook needs a trained UNet model. We provide a pre-trained
model, trained for 20 epochs with the full
`Kits-19 `__ frames dataset, which
has an F1 score on the validation set of 0.9. The training code is
-available in `this notebook `__.
+available in `this notebook `__.
NNCF for PyTorch models requires a C++ compiler. On Windows, install
`Microsoft Visual Studio
@@ -88,9 +88,9 @@ Table of contents:
.. code:: ipython3
import platform
-
+
%pip install -q "openvino>=2023.3.0" "monai>=0.9.1" "torchmetrics>=0.11.0" "nncf>=2.8.0" "opencv-python" torch tqdm --extra-index-url https://download.pytorch.org/whl/cpu
-
+
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
@@ -100,10 +100,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -122,9 +118,9 @@ Imports
import zipfile
from pathlib import Path
from typing import Union
-
+
warnings.filterwarnings("ignore", category=UserWarning)
-
+
import cv2
import matplotlib.pyplot as plt
import monai
@@ -135,29 +131,31 @@ Imports
from monai.transforms import LoadImage
from nncf.common.logging.logger import set_log_level
from torchmetrics import F1Score as F1
-
- set_log_level(logging.ERROR) # Disables all NNCF info and warning messages
-
- from custom_segmentation import SegmentationModel
- from async_pipeline import show_live_inference
-
- # Fetch `notebook_utils` module
import requests
-
+
+
+ set_log_level(logging.ERROR) # Disables all NNCF info and warning messages
+
+ # Fetch `notebook_utils` module
r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py")
+ open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
+ if not Path("./custom_segmentation.py").exists():
+ download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/ct-segmentation-quantize/custom_segmentation.py")
+ from custom_segmentation import SegmentationModel
+
+ if not Path("./async_pipeline.py").exists():
+ download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/ct-segmentation-quantize/async_pipeline.py")
+ from async_pipeline import show_live_inference
+
.. parsed-literal::
- 2024-04-17 23:29:37.722637: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:29:37.758897: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-06 23:48:33.144412: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-06 23:48:33.181396: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:29:38.341773: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-06 23:48:33.764576: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@@ -171,9 +169,7 @@ Settings
By default, this notebook will download one CT scan from the KITS19
-dataset that will be used for quantization. To use the full dataset, set
-``BASEDIR`` to the path of the dataset, as prepared according to the
-`Data Preparation `__ notebook.
+dataset that will be used for quantization.
.. code:: ipython3
@@ -201,13 +197,13 @@ notebook `__.
state_dict_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/kidney-segmentation-kits19/unet_kits19_state_dict.pth"
state_dict_file = download_file(state_dict_url, directory="pretrained_model")
state_dict = torch.load(state_dict_file, map_location=torch.device("cpu"))
-
+
new_state_dict = {}
for k, v in state_dict.items():
new_key = k.replace("_model.", "")
new_state_dict[new_key] = v
new_state_dict.pop("loss_function.pos_weight")
-
+
model = monai.networks.nets.BasicUNet(spatial_dims=2, in_channels=1, out_channels=1).eval()
model.load_state_dict(new_state_dict)
@@ -252,9 +248,15 @@ Download CT-scan Data
print(f"Data for case_{CASE:05d} exists")
+
.. parsed-literal::
- Data for case_00117 exists
+ case_00117.zip: 0%| | 0.00/5.48M [00:00 5000)
# Remove extra image dimension and rotate and flip the image for visualization
image = rotate_and_flip(image_data.squeeze())
-
+
# The data loader returns annotations as (index, mask) and mask in shape (H,W)
mask = rotate_and_flip(mask)
-
+
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
ax[0].imshow(image, cmap="gray")
ax[1].imshow(mask, cmap="gray");
@@ -420,7 +422,7 @@ this notebook.
.. code:: ipython3
fp32_ir_path = MODEL_DIR / Path("unet_kits19_fp32.xml")
-
+
fp32_ir_model = ov.convert_model(model, example_input=torch.ones(1, 1, 512, 512, dtype=torch.float32))
ov.save_model(fp32_ir_model, str(fp32_ir_path))
@@ -433,11 +435,7 @@ this notebook.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
@@ -469,8 +467,8 @@ steps:
"""
images, _ = data_item
return images
-
-
+
+
data_loader = torch.utils.data.DataLoader(dataset)
calibration_dataset = nncf.Dataset(data_loader, transform_fn)
quantized_model = nncf.quantize(
@@ -536,22 +534,18 @@ Convert quantized model to OpenVINO IR model and save it.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:337: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:337: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_low.item()
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:345: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/torch/quantization/layers.py:345: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
return self._level_high.item()
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/monai/networks/nets/basic_unet.py:168: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x_e.shape[-i - 1] != x_0.shape[-i - 1]:
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1102: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:1116: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!
-
- Mismatched elements: 248412 / 262144 (94.8%)
- Greatest absolute difference: 3.334601879119873 at index (0, 0, 345, 29) (up to 1e-05 allowed)
- Greatest relative difference: 21800.240266957342 at index (0, 0, 242, 213) (up to 1e-05 allowed)
+
+ Mismatched elements: 249823 / 262144 (95.3%)
+ Greatest absolute difference: 4.744992733001709 at index (0, 0, 242, 231) (up to 1e-05 allowed)
+ Greatest relative difference: 26823.613314473136 at index (0, 0, 124, 22) (up to 1e-05 allowed)
_check_trace(
@@ -576,7 +570,7 @@ Compare File Size
fp32_ir_model_size = fp32_ir_path.with_suffix(".bin").stat().st_size / 1024
quantized_model_size = int8_ir_path.with_suffix(".bin").stat().st_size / 1024
-
+
print(f"FP32 IR model size: {fp32_ir_model_size:.2f} KB")
print(f"INT8 model size: {quantized_model_size:.2f} KB")
@@ -584,7 +578,7 @@ Compare File Size
.. parsed-literal::
FP32 IR model size: 3864.14 KB
- INT8 model size: 1940.40 KB
+ INT8 model size: 1953.48 KB
Select Inference Device
@@ -597,16 +591,16 @@ Select Inference Device
core = ov.Core()
# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.
device_list = ["MULTI:CPU,GPU" if "GPU" in core.available_devices else "AUTO"]
-
+
import ipywidgets as widgets
-
+
device = widgets.Dropdown(
options=core.available_devices + device_list,
value=device_list[0],
description="Device:",
disabled=False,
)
-
+
device
@@ -627,7 +621,7 @@ Compare Metrics for the original model and the quantized model to be sure that t
int8_compiled_model = core.compile_model(int8_ir_model, device.value)
int8_f1 = compute_f1(int8_compiled_model, dataset)
-
+
print(f"FP32 F1: {fp32_f1:.3f}")
print(f"INT8 F1: {int8_f1:.3f}")
@@ -674,18 +668,18 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
+ [ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ]
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
+ [ INFO ]
+ [ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
- [ INFO ] Read model took 8.73 ms
+ [ INFO ] Read model took 8.87 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [?,?,?,?]
@@ -699,11 +693,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [?,1,16..,16..]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 151.10 ms
+ [ INFO ] Compile model took 148.60 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@@ -724,6 +714,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
@@ -733,12 +724,13 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ ERROR ] Input x is dynamic. Provide data shapes!
Traceback (most recent call last):
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 486, in main
+ File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 486, in main
data_queue = get_input_data(paths_to_input, app_inputs_info)
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
+ File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/utils/inputs_filling.py", line 123, in get_input_data
raise Exception(f"Input {info.name} is dynamic. Provide data shapes!")
Exception: Input x is dynamic. Provide data shapes!
@@ -755,22 +747,18 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
+ [ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ]
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
+ [ INFO ]
+ [ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 13.30 ms
+ [ INFO ] Read model took 10.46 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] x (node: x) : f32 / [...] / [1,1,512,512]
@@ -784,11 +772,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: __module.final_conv/aten::_convolution/Add) : f32 / [...] / [1,1,512,512]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 233.12 ms
+ [ INFO ] Compile model took 253.55 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model49
@@ -809,6 +793,7 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model49
[ INFO ] NUM_STREAMS: 1
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
@@ -818,30 +803,23 @@ be run in the notebook with ``! benchmark_app`` or
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
- [ INFO ] Fill input 'x' with random values
+ [ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference synchronously, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 31.47 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 28.04 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 965 iterations
- [ INFO ] Duration: 15011.79 ms
+ [ INFO ] Count: 969 iterations
+ [ INFO ] Duration: 15000.48 ms
[ INFO ] Latency:
- [ INFO ] Median: 15.31 ms
- [ INFO ] Average: 15.36 ms
- [ INFO ] Min: 15.02 ms
- [ INFO ] Max: 17.48 ms
- [ INFO ] Throughput: 64.28 FPS
+ [ INFO ] Median: 15.24 ms
+ [ INFO ] Average: 15.28 ms
+ [ INFO ] Min: 14.97 ms
+ [ INFO ] Max: 17.08 ms
+ [ INFO ] Throughput: 64.60 FPS
Visually Compare Inference Results
@@ -875,11 +853,11 @@ seed is displayed to enable reproducing specific runs of this cell.
# to binary segmentation masks
def sigmoid(x):
return np.exp(-np.logaddexp(0, -x))
-
-
+
+
num_images = 4
colormap = "gray"
-
+
# Load FP32 and INT8 models
core = ov.Core()
fp_model = core.read_model(fp32_ir_path)
@@ -888,18 +866,18 @@ seed is displayed to enable reproducing specific runs of this cell.
compiled_model_int8 = core.compile_model(int8_model, device_name=device.value)
output_layer_fp = compiled_model_fp.output(0)
output_layer_int8 = compiled_model_int8.output(0)
-
+
# Create subset of dataset
background_slices = (item for item in dataset if np.count_nonzero(item[1]) == 0)
kidney_slices = (item for item in dataset if np.count_nonzero(item[1]) > 50)
data_subset = random.sample(list(background_slices), 2) + random.sample(list(kidney_slices), 2)
-
+
# Set seed to current time. To reproduce specific results, copy the printed seed
# and manually set `seed` to that value.
seed = int(time.time())
random.seed(seed)
print(f"Visualizing results with seed {seed}")
-
+
fig, ax = plt.subplots(nrows=num_images, ncols=4, figsize=(24, num_images * 4))
for i, (image, mask) in enumerate(data_subset):
display_image = rotate_and_flip(image.squeeze())
@@ -908,13 +886,13 @@ seed is displayed to enable reproducing specific runs of this cell.
input_image = np.expand_dims(image, 0)
res_fp = compiled_model_fp([input_image])
res_int8 = compiled_model_int8([input_image])
-
+
# Process inference outputs and convert to binary segementation masks
result_mask_fp = sigmoid(res_fp[output_layer_fp]).squeeze().round().astype(np.uint8)
result_mask_int8 = sigmoid(res_int8[output_layer_int8]).squeeze().round().astype(np.uint8)
result_mask_fp = rotate_and_flip(result_mask_fp)
result_mask_int8 = rotate_and_flip(result_mask_int8)
-
+
# Display images, annotations, FP32 result and INT8 result
ax[i, 0].imshow(display_image, cmap=colormap)
ax[i, 1].imshow(target_mask, cmap=colormap)
@@ -926,7 +904,7 @@ seed is displayed to enable reproducing specific runs of this cell.
.. parsed-literal::
- Visualizing results with seed 1713389447
+ Visualizing results with seed 1715032183
@@ -968,7 +946,7 @@ overlay of the segmentation mask on the original image/frame.
.. code:: ipython3
CASE = 117
-
+
segmentation_model = SegmentationModel(ie=core, model_path=int8_ir_path, sigmoid=True, rotate_and_flip=True)
case_path = BASEDIR / f"case_{CASE:05d}"
image_paths = sorted(case_path.glob("imaging_frames/*jpg"))
@@ -1009,8 +987,8 @@ performs inference, and displays the results on the frames loaded in
.. parsed-literal::
- Loaded model to AUTO in 0.22 seconds.
- Total time for 68 frames: 2.70 seconds, fps:25.59
+ Loaded model to AUTO in 0.23 seconds.
+ Total time for 68 frames: 2.67 seconds, fps:25.86
References
diff --git a/docs/notebooks/ct-segmentation-quantize-nncf-with-output_files/ct-segmentation-quantize-nncf-with-output_37_1.png b/docs/notebooks/ct-segmentation-quantize-nncf-with-output_files/ct-segmentation-quantize-nncf-with-output_37_1.png
index 156f435eba7..51147de15b1 100644
--- a/docs/notebooks/ct-segmentation-quantize-nncf-with-output_files/ct-segmentation-quantize-nncf-with-output_37_1.png
+++ b/docs/notebooks/ct-segmentation-quantize-nncf-with-output_files/ct-segmentation-quantize-nncf-with-output_37_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:08f7f46aea3e4c64f7bcb4dfe777299e9cb77cf5ec36396e340c1be6e8206aef
-size 382874
+oid sha256:16b3bd0c7d257596831f1d01b363b5f52edfd6db56c147651414f6ac0f77d958
+size 382301
diff --git a/docs/notebooks/decidiffusion-image-generation-with-output.rst b/docs/notebooks/decidiffusion-image-generation-with-output.rst
index 07a0f1a2896..9e878c71157 100644
--- a/docs/notebooks/decidiffusion-image-generation-with-output.rst
+++ b/docs/notebooks/decidiffusion-image-generation-with-output.rst
@@ -89,7 +89,7 @@ install required packages
.. code:: ipython3
- %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "diffusers" "transformers" "torch>=2.1" "pillow" "openvino>=2023.1.0" "gradio>=4.19" "datasets>=2.14.6" "huggingface-hub>=0.19.4" "nncf>=2.7.0" "peft==0.6.2"
+ %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "diffusers" "transformers" "torch>=2.1" "pillow" "openvino>=2023.1.0" "gradio>=4.19" "datasets>=2.14.6" "huggingface-hub>=0.19.4" "nncf>=2.7.0" "peft==0.6.2" "opencv-python"
Prepare DeciDiffusion models for OpenVINO format conversion
-----------------------------------------------------------
diff --git a/docs/notebooks/deep-floyd-if-convert-with-output.rst b/docs/notebooks/deep-floyd-if-convert-with-output.rst
deleted file mode 100644
index 499c29a6934..00000000000
--- a/docs/notebooks/deep-floyd-if-convert-with-output.rst
+++ /dev/null
@@ -1,1061 +0,0 @@
-Image generation with DeepFloyd IF and OpenVINO™
-================================================
-
-DeepFloyd IF is an advanced open-source text-to-image model that
-delivers remarkable photorealism and language comprehension. DeepFloyd
-IF consists of a frozen text encoder and three cascaded pixel diffusion
-modules: a base model that creates 64x64 pixel images based on text
-prompts and two super-resolution models, each designed to generate
-images with increasing resolution: 256x256 pixel and 1024x1024 pixel.
-All stages of the model employ a frozen text encoder, built on the T5
-transformer, to derive text embeddings, which are then passed to a UNet
-architecture enhanced with cross-attention and attention pooling.
-
-Text encoder impact
-~~~~~~~~~~~~~~~~~~~
-
-- **Profound text prompt comprehension.** The generation pipeline
- leverages the T5-XXL-1.1 Large Language Model (LLM) as a text
- encoder. Its intelligence is backed by a substantial number of
- text-image cross-attention layers, this ensures superior alignment
- between the prompt and the generated image.
-
-- **Realistic text in generated images.** Capitalizing on the
- capabilities of the T5 model, DeepFloyd IF produces readable text
- depictions alongside objects with distinct attributes, which have
- typically been a challenge for most existing text-to-image models.
-
-DeepFloyd IF Distinctive Features
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-First of all, it is **Modular**. DeepFloyd IF pipeline is a consecutive
-inference of several neural networks.
-
-Which makes it **Cascaded**. The base model generates low-resolution
-samples, then super-resolution models upsample the images to produce
-high-resolution results. The models were individually trained at
-different resolutions.
-
-DeepFloyd IF employs **Diffusion** models. Diffusion models are machine
-learning systems that are trained to denoise random Gaussian noise step
-by step, to get to a sample of interest, such as an image. Diffusion
-models have been shown to achieve state-of-the-art results for
-generating image data.
-
-And finally, DeepFloyd IF operates in **Pixel** space. Unlike latent
-diffusion models (Stable Diffusion for instance), the diffusion is
-implemented on a pixel level.
-
-.. figure:: https://github.com/deep-floyd/IF/raw/develop/pics/deepfloyd_if_scheme.jpg
- :alt: deepfloyd_if_scheme
-
- deepfloyd_if_scheme
-
-The graph above depicts the three-stage generation pipeline: A text
-prompt is passed through the frozen T5-XXL LLM to convert it into a
-vector in embedded space.
-
-1. Stage 1: The first diffusion model in the cascade transforms the
- embedding vector into a 64x64 image. The DeepFloyd team has trained
- **three versions** of the base model, each with different parameters:
- IF-I 400M, IF-I 900M, and IF-I 4.3B. The smallest one is used by
- default, but users are free to change the checkpoint name to
- `“DeepFloyd/IF-I-L-v1.0” `__
- or
- `“DeepFloyd/IF-I-XL-v1.0” `__
-
-2. Stage 2: To upscale the image, two text-conditional super-resolution
- models (Efficient U-Net) are applied to the output of the first
- diffusion model. The first of these upscales the sample from 64x64
- pixel to 256x256 pixel resolution. Again, several versions of this
- model are available: IF-II 400M (default) and IF-II 1.2B (checkpoint
- name “DeepFloyd/IF-II-L-v1.0”).
-
-3. Stage 3: Follows the same path as Stage 2 and upscales the image to
- 1024x1024 pixel resolution. It is not released yet, so we will use a
- conventional Super Resolution network to get hi-res results.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Prerequisites <#prerequisites>`__
-
- - `Authentication <#authentication>`__
-
-- `DeepFloyd IF in Diffusers
- library <#deepfloyd-if-in-diffusers-library>`__
-- `Convert models to OpenVINO Intermediate representation (IR)
- format <#convert-models-to-openvino-intermediate-representation-ir-format>`__
-- `1. Convert Text Encoder <#1--convert-text-encoder>`__
-- `Convert the first Pixel Diffusion module’s
- UNet <#convert-the-first-pixel-diffusion-modules-unet>`__
-- `Convert the second pixel diffusion
- module <#convert-the-second-pixel-diffusion-module>`__
-- `Prepare Inference pipeline <#prepare-inference-pipeline>`__
-
- - `Select inference device <#select-inference-device>`__
-
-- `Run Text-to-Image generation <#run-text-to-image-generation>`__
-
- - `Text Encoder inference <#text-encoder-inference>`__
- - `First Stage diffusion block
- inference <#first-stage-diffusion-block-inference>`__
- - `Second Stage diffusion block
- inference <#second-stage-diffusion-block-inference>`__
- - `Third Stage diffusion block <#third-stage-diffusion-block>`__
- - `Upscale the generated image using a Super Resolution
- network <#upscale-the-generated-image-using-a-super-resolution-network>`__
-
- - `Download the Super Resolution model
- weights <#download-the-super-resolution-model-weights>`__
- - `Reshape the model’s inputs <#reshape-the-models-inputs>`__
- - `Prepare the input images and run the
- model <#prepare-the-input-images-and-run-the-model>`__
- - `Display the result <#display-the-result>`__
-
-- `Try out the converted pipeline with
- Gradio <#try-out-the-converted-pipeline-with-gradio>`__
-- `Next steps <#next-steps>`__
-
- **NOTE**:
-
-..
-
- - *This example requires the download of roughly 27 GB of model
- checkpoints, which could take some time depending on your internet
- connection speed. Additionally, the converted models will consume
- another 27 GB of disk space.*
- - *Please be aware that a minimum of 32 GB of RAM is necessary to
- convert and run inference on the models. There may be instances
- where the notebook appears to freeze or stop responding.*
- - *To access the model checkpoints, you’ll need a Hugging Face
- account. You’ll also be prompted to explicitly accept the*\ `model
- license `__\ *.*
-
-Prerequisites
--------------
-
-
-
-Install required packages.
-
-.. code:: ipython3
-
- # Set up requirements
-
- %pip install -q --upgrade pip
- %pip install -q "torch>2.1" transformers "diffusers>=0.16.1" accelerate safetensors sentencepiece huggingface_hub --extra-index-url https://download.pytorch.org/whl/cpu
- %pip install -q "openvino>=2023.3.0" opencv-python tqdm
- %pip install -q "gradio>=4.19"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
- DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- Note: you may need to restart the kernel to use updated packages.
- DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- Note: you may need to restart the kernel to use updated packages.
- DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- import gc
- from pathlib import Path
-
- from PIL import Image
- from diffusers import DiffusionPipeline
- import openvino as ov
- import torch
- from utils import (
- TextEncoder,
- UnetFirstStage,
- UnetSecondStage,
- convert_result_to_image,
- download_omz_model,
- )
-
-.. code:: ipython3
-
- checkpoint_variant = "fp16"
- model_dtype = torch.float32
- ir_input_type = ov.Type.f32
- compress_to_fp16 = True
-
- models_dir = Path("./models")
- models_dir.mkdir(exist_ok=True)
-
- encoder_ir_path = models_dir / "encoder_ir.xml"
- first_stage_unet_ir_path = models_dir / "unet_ir_I.xml"
- second_stage_unet_ir_path = models_dir / "unet_ir_II.xml"
-
-
- def pt_to_pil(images):
- """
- Convert a torch image to a PIL image.
- """
- images = (images / 2 + 0.5).clamp(0, 1)
- images = images.cpu().permute(0, 2, 3, 1).float().numpy()
- images = numpy_to_pil(images)
- return images
-
-
- def numpy_to_pil(images):
- """
- Convert a numpy image or a batch of images to a PIL image.
- """
- if images.ndim == 3:
- images = images[None, ...]
- images = (images * 255).round().astype("uint8")
- if images.shape[-1] == 1:
- # special case for grayscale (single channel) images
- pil_images = [Image.fromarray(image.squeeze(), mode="L") for image in images]
- else:
- pil_images = [Image.fromarray(image) for image in images]
-
- return pil_images
-
-Authentication
-~~~~~~~~~~~~~~
-
-
-
-In order to access IF checkpoints, users need to provide an
-authentication token.
-
-If you already have a token, you can input it into the provided form in
-the next cell. If not, please proceed according to the following
-instructions:
-
-1. Make sure to have a `Hugging Face `__
- account and be logged in
-2. Accept the license on the model card of
- `DeepFloyd/IF-I-M-v1.0 `__
-3. To generate a token, proceed to `this
- page `__
-
-Uncheck the ``Add token as git credential?`` box.
-
-.. code:: ipython3
-
- from huggingface_hub import login
-
- # Execute this cell to access the authentication form
- login()
-
-
-
-.. parsed-literal::
-
- VBox(children=(HTML(value='
`__. Diffusers package
-exposes the ``DiffusionPipeline`` class, simplifying experiments with
-diffusion models. The code below demonstrates how to create a
-``DiffusionPipeline`` using IF configs:
-
-.. code:: ipython3
-
- # Downloading the model weights may take some time. The approximate total checkpoints size is 27GB.
- stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant=checkpoint_variant, torch_dtype=model_dtype)
-
- stage_2 = DiffusionPipeline.from_pretrained(
- "DeepFloyd/IF-II-M-v1.0",
- text_encoder=None,
- variant=checkpoint_variant,
- torch_dtype=model_dtype,
- )
-
-
-
-.. parsed-literal::
-
- model_index.json: 0%| | 0.00/604 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
-
- A mixture of fp16 and non-fp16 filenames will be loaded.
- Loaded fp16 filenames:
- [unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors, safety_checker/model.fp16.safetensors]
- Loaded non-fp16 filenames:
- [watermarker/diffusion_pytorch_model.safetensors
- If this behavior is not expected, please check your folder structure.
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00= 64:
- /home/ea/.local/lib/python3.8/site-packages/diffusers/models/resnet.py:173: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- assert hidden_states.shape[1] == self.channels
- /home/ea/.local/lib/python3.8/site-packages/diffusers/models/resnet.py:186: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if hidden_states.shape[0] >= 64:
-
-
-Convert the second pixel diffusion module
------------------------------------------
-
-
-
-The second Diffusion module in the cascade generates 256x256 pixel
-images.
-
-The second stage pipeline will use bilinear interpolation to upscale the
-64x64 image that was generated in the previous stage to a higher 256x256
-resolution. Then it will denoise the image taking into account the
-encoded user prompt.
-
-.. code:: ipython3
-
- if not second_stage_unet_ir_path.exists():
- unet_2_ir = ov.convert_model(
- stage_2.unet,
- example_input=(
- torch.ones(2, 6, 256, 256),
- torch.tensor(1).int(),
- torch.ones(2, 77, 4096),
- torch.ones(2).int(),
- ),
- input=[
- ((2, 6, 256, 256), ir_input_type),
- ((), ov.Type.i32),
- ((2, 77, 4096), ir_input_type),
- ((2,), ov.Type.i32),
- ],
- )
-
- ov.save_model(unet_2_ir, second_stage_unet_ir_path, compress_to_fp16=compress_to_fp16)
-
- del unet_2_ir
-
- stage_2_config = stage_2.unet.config
- del stage_2.unet
-
- gc.collect();
-
-Prepare Inference pipeline
---------------------------
-
-
-
-The original pipeline from the source repository will be reused in this
-example. In order to achieve this, adapter classes were created to
-enable OpenVINO models to replace Pytorch models and integrate
-seamlessly into the pipeline.
-
-.. code:: ipython3
-
- core = ov.Core()
-
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-Run Text-to-Image generation
-----------------------------
-
-
-
-Now, we can set a text prompt for image generation and execute the
-inference pipeline. Optionally, you can also modify the random generator
-seed for latent state initialization and adjust the number of images to
-be generated for the given prompt.
-
-Text Encoder inference
-~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Inferring model in FP16 on GPU significantly increases performance and
-reduces required memory, but it may lead to lesser accuracy on some
-models. Particularly stage_1.text_encoder suffers from that. To avoid
-this effect we should calibrate some layer’s precision,
-partially_upcast_nodes_to_fp32 helper upcasts only most
-precision-sensitive nodes to FP32 so that accuracy is restored while
-most of the nodes remain in FP16. With that inference performance
-remains comparable with full FP16.
-
-.. code:: ipython3
-
- # Fetch `model_upcast_utils` which helps to restore accuracy when inferred on GPU
- import requests
-
- r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/model_upcast_utils.py")
- open("model_upcast_utils.py", "w").write(r.text)
-
- from model_upcast_utils import (
- is_model_partially_upcasted,
- partially_upcast_nodes_to_fp32,
- )
-
- encoder_ov_model = core.read_model(encoder_ir_path)
- if "GPU" in core.available_devices and not is_model_partially_upcasted(encoder_ov_model):
- example_input_prompt = "ultra close color photo portrait of rainbow owl with deer horns in the woods"
- text_inputs = stage_1.tokenizer(example_input_prompt, max_length=77, padding="max_length", return_tensors="np")
- upcasted_ov_model = partially_upcast_nodes_to_fp32(
- encoder_ov_model,
- text_inputs.input_ids,
- upcast_ratio=0.05,
- operation_types=["MatMul"],
- batch_size=10,
- )
- del encoder_ov_model
- gc.collect()
-
- import os
-
- os.remove(encoder_ir_path)
- os.remove(str(encoder_ir_path).replace(".xml", ".bin"))
- ov.save_model(upcasted_ov_model, encoder_ir_path, compress_to_fp16=compress_to_fp16)
-
-.. code:: ipython3
-
- prompt = "ultra close color photo portrait of rainbow owl with deer horns in the woods"
- negative_prompt = "blurred unreal uncentered occluded"
-
- # Initialize TextEncoder wrapper class
- stage_1.text_encoder = TextEncoder(encoder_ir_path, dtype=model_dtype, device=device.value)
- print("The model has been loaded")
-
- # Generate text embeddings
- prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
-
- # Delete the encoder to free up memory
- del stage_1.text_encoder.encoder_openvino
- gc.collect()
-
-
-.. parsed-literal::
-
- The model has been loaded
-
-
-.. parsed-literal::
-
- /home/ea/.local/lib/python3.8/site-packages/diffusers/configuration_utils.py:135: FutureWarning: Accessing config attribute `unet` directly via 'IFPipeline' object attribute is deprecated. Please access 'unet' over 'IFPipeline's config object instead, e.g. 'scheduler.config.unet'.
- deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
-
-
-.. parsed-literal::
-
- CPU times: user 32.9 s, sys: 6.82 s, total: 39.8 s
- Wall time: 11.2 s
-
-
-
-
-.. parsed-literal::
-
- 0
-
-
-
-First Stage diffusion block inference
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- # Changing the following parameters will affect the model output
- # Note that increasing the number of diffusion steps will increase the inference time linearly.
- RANDOM_SEED = 42
- N_DIFFUSION_STEPS = 50
-
- # Initialize the First Stage UNet wrapper class
- stage_1.unet = UnetFirstStage(first_stage_unet_ir_path, stage_1_config, dtype=model_dtype, device=device.value)
- print("The model has been loaded")
-
- # Fix PRNG seed
- generator = torch.manual_seed(RANDOM_SEED)
-
- # Inference
- image = stage_1(
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=N_DIFFUSION_STEPS,
- ).images
-
- # Delete the model to free up memory
- del stage_1.unet.unet_openvino
- gc.collect()
-
- # Show the image
- pt_to_pil(image)[0]
-
-
-.. parsed-literal::
-
- The model has been loaded
-
-
-
-.. parsed-literal::
-
- 0%| | 0/50 [00:00`__
-to enhance our low-resolution result!
-
-Note, this step will be substituted with the Third IF stage upon its
-release!
-
-.. code:: ipython3
-
- import platform
-
- if platform.system() != "Windows":
- %pip install -q "matplotlib>=3.4"
- else:
- %pip install -q "matplotlib>=3.4,<3.7"
-
-
-.. parsed-literal::
-
- DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
- Note: you may need to restart the kernel to use updated packages.
-
-
-Download the Super Resolution model weights
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-.. code:: ipython3
-
- import cv2
- import numpy as np
-
- # 1032: 4x superresolution, 1033: 3x superresolution
- model_name = "single-image-super-resolution-1032"
- download_omz_model(model_name, models_dir)
-
- sr_model_xml_path = models_dir / f"{model_name}.xml"
-
-
-
-.. parsed-literal::
-
- models/single-image-super-resolution-1032.xml: 0%| | 0.00/89.3k [00:00`__
-
-.. code:: ipython3
-
- # Build up the pipeline
- stage_1.text_encoder = TextEncoder(encoder_ir_path, dtype=model_dtype, device=device.value)
- print("The model has been loaded")
-
- stage_1.unet = UnetFirstStage(first_stage_unet_ir_path, stage_1_config, dtype=model_dtype, device=device.value)
- print("The Stage-1 UNet has been loaded")
-
- stage_2.unet = UnetSecondStage(second_stage_unet_ir_path, stage_2_config, dtype=model_dtype, device=device.value)
- print("The Stage-2 UNet has been loaded")
-
- generator = torch.manual_seed(RANDOM_SEED)
-
-
- # Combine the models' calls into a single `_generate` function
- def _generate(prompt, negative_prompt):
- # Text encoder inference
- prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
- # Stage-1 UNet Inference
- image = stage_1(
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=N_DIFFUSION_STEPS,
- ).images
- # Stage-2 UNet Inference
- image = stage_2(
- image=image,
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=20,
- ).images
- # Infer Super Resolution model
- original_image = np.array(pt_to_pil(image)[0])
- bicubic_image = cv2.resize(src=original_image, dsize=(1024, 1024), interpolation=cv2.INTER_CUBIC)
- # Reshape the images from (H,W,C) to (N,C,H,W) as expected by the model.
- input_image_original = np.expand_dims(original_image.transpose(2, 0, 1), axis=0)
- input_image_bicubic = np.expand_dims(bicubic_image.transpose(2, 0, 1), axis=0)
- # Model Inference
- result = compiled_sr_model([input_image_original, input_image_bicubic])[compiled_sr_model.output(0)]
- return convert_result_to_image(result)
-
-
-.. parsed-literal::
-
- The model has been loaded
- The Stage-1 UNet has been loaded
- The Stage-2 UNet has been loaded
-
-
-.. code:: ipython3
-
- import gradio as gr
-
- demo = gr.Interface(
- _generate,
- inputs=[
- gr.Textbox(label="Text Prompt"),
- gr.Textbox(label="Negative Text Prompt"),
- ],
- outputs=["image"],
- examples=[
- [
- "ultra close color photo portrait of rainbow owl with deer horns in the woods",
- "blurred unreal uncentered occluded",
- ],
- [
- "A scaly mischievous dragon is driving the car in a street art style",
- "blurred uncentered occluded",
- ],
- ],
- )
- try:
- demo.queue().launch(debug=False)
- except Exception:
- demo.queue().launch(share=True, debug=False)
-
- # If you are launching remotely, specify server_name and server_port
- # EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`
- # To learn more please refer to the Gradio docs: https://gradio.app/docs/
-
-Next steps
-----------
-
-
-
-Open the `deep-floyd-if-optimize `__
-notebook to quantize stage 1 and stage 2 U-Net models with the
-Post-training Quantization API of NNCF and compress weights of the text
-encoder. Then compare the converted and optimized OpenVINO models.
diff --git a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_29_3.png b/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_29_3.png
deleted file mode 100644
index 5a6b37017dd..00000000000
--- a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_29_3.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f8a26ce14426df514d34e2a3722ad5c94bea8c89dfe95b66d82ac1efa1e1e7e5
-size 10887
diff --git a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_31_3.png b/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_31_3.png
deleted file mode 100644
index 04b33261e83..00000000000
--- a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_31_3.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7d5a3615f47a2ff2a0f4d537a4e91556c6cedf1fb213056618c072c9ff8f0979
-size 129741
diff --git a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_41_0.png b/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_41_0.png
deleted file mode 100644
index 96c991d0dac..00000000000
--- a/docs/notebooks/deep-floyd-if-convert-with-output_files/deep-floyd-if-convert-with-output_41_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:bab7f183b691aae5fae3d4cd085d8b1a479e128a52778a91da76f992fcebdade
-size 1349882
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output.rst b/docs/notebooks/deep-floyd-if-optimize-with-output.rst
deleted file mode 100644
index 89d976895e2..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output.rst
+++ /dev/null
@@ -1,839 +0,0 @@
-Post-Training Quantization and Weights Compression of DeepFloyd IF model with NNCF
-==================================================================================
-
-The goal of this tutorial is to demonstrate how to speed up the model by
-applying 8-bit post-training quantization and weights compression from
-`NNCF `__ (Neural Network
-Compression Framework) and infer optimized model via OpenVINO™ Toolkit.
-
- **NOTE**: you should run
- `deep-floyd-if-convert `__ notebook
- first to generate OpenVINO IR model that is used for optimization.
-
-The optimization process contains the following steps: 1. Compress
-weights of the converted OpenVINO text encoder from
-`notebook `__ with NNCF. 2. Quantize the
-converted stage_1 and stage_2 U-Nets from
-`notebook `__ with NNCF. 2. Check the model
-result using the same input data from the
-`notebook `__. 3. Compare model size of
-converted and optimized models. 4. Compare performance of converted and
-optimized models.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Prerequisites <#prerequisites>`__
-- `Compress weights <#compress-weights>`__
-- `Quantize <#quantize>`__
-
- - `Prepare dataset <#prepare-dataset>`__
- - `Quantize first stage U-Net <#quantize-first-stage-u-net>`__
- - `Quantize second stage U-Net <#quantize-second-stage-u-net>`__
-
-- `Run optimized OpenVINO model <#run-optimized-openvino-model>`__
-
- - `Compare file sizes <#compare-file-sizes>`__
- - `Compare performance time of the converted and optimized
- models <#compare-performance-time-of-the-converted-and-optimized-models>`__
-
-Prerequisites
--------------
-
-
-
-.. code:: ipython3
-
- %pip install -q datasets "nncf>=2.6.0" "torch>=2.1" tqdm
-
-.. code:: ipython3
-
- import nncf
- import torch
- import openvino as ov
-
- from diffusers import DiffusionPipeline
- from diffusers.utils.pil_utils import pt_to_pil
- from pathlib import Path
- from typing import Any, List
-
- from utils import TextEncoder, UnetFirstStage, UnetSecondStage
-
- checkpoint_variant = "fp16"
- model_dtype = torch.float32
- RANDOM_SEED = 42
- N_DIFFUSION_STEPS = 50
- UNET_2_STEPS = 20
-
- core = ov.Core()
-
-
-.. parsed-literal::
-
- INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
-
-
-.. code:: ipython3
-
- MODEL_DIR = Path("./models")
- TEXT_ENCODER_IR_PATH = MODEL_DIR / "encoder_ir.xml"
- UNET_I_IR_PATH = MODEL_DIR / "unet_ir_I.xml"
- UNET_II_IR_PATH = MODEL_DIR / "unet_ir_II.xml"
-
- if not (TEXT_ENCODER_IR_PATH.exists() and UNET_I_IR_PATH.exists() and UNET_II_IR_PATH.exists()):
- raise RuntimeError("This notebook should be run after deep-floyd-if notebook")
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')
-
-
-
-Compress weights
-----------------
-
-
-
-Text encoder model consumes ~22 GB of disk space. To avoid running out
-of memory, we suggest using 8-bit weights compression instead of
-quantization. An optimized model will show less speed up than a
-quantized model, but this will significantly reduce the model footprint.
-
-.. code:: ipython3
-
- %%time
-
- text_encoder = core.read_model(TEXT_ENCODER_IR_PATH)
- text_encoder_optimized = nncf.compress_weights(text_encoder)
-
- TEXT_ENCODER_INT8_IR_PATH = Path("_optimized.".join(TEXT_ENCODER_IR_PATH.as_posix().split(".")))
- ov.save_model(text_encoder_optimized, TEXT_ENCODER_INT8_IR_PATH)
-
-
-.. parsed-literal::
-
- 2023-10-30 08:36:34.384792: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2023-10-30 08:36:34.423283: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
- To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
- 2023-10-30 08:36:35.184200: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- CPU times: user 3min 16s, sys: 58 s, total: 4min 14s
- Wall time: 4min 12s
-
-
-Quantize
---------
-
-
-
-Prepare dataset
-~~~~~~~~~~~~~~~
-
-
-
-DeepFloyd IF consists of a U-Net model for first and second stages.
-First stage U-Net generates 64x64 px image based on text prompt, second
-stage U-Net generates a 256x256 px image based on image from previous
-step. We use a portion of train
-`LAION2B `__
-dataset from Hugging Face as calibration data. LAION2B is the English
-subset of the `LAION5B `__ dataset,
-contains over 2 billion objects.
-
-.. code:: ipython3
-
- import numpy as np
- from datasets import load_dataset
-
- np.random.seed(RANDOM_SEED)
-
-
- def get_negative_prompt():
- negative_prompts = [
- "amateur",
- "blurred",
- "deformed",
- "disfigured",
- "disgusting",
- "jpeg artifacts",
- "low contrast",
- "low quality",
- "low saturation",
- "mangled",
- "morbid",
- "mutilated",
- "mutation",
- "out of frame",
- "out of frame",
- "ugly",
- "uncentered",
- "underexposed",
- "unreal",
- ]
- num_elements = np.random.randint(2, 6)
- random_elements = np.random.choice(negative_prompts, num_elements)
- return [" ".join(random_elements)]
-
-
- def prepare_calibration_data(dataloader, stage_1):
- """
- This function prepares calibration data from a dataloader for a specified number of initialization steps.
- It iterates over the dataloader, fetching batches and storing the relevant data.
- """
- data = []
- for batch in dataloader:
- prompt = batch["TEXT"]
- negative_prompt = get_negative_prompt()
- prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
- data.append((prompt_embeds, negative_embeds))
- return data
-
-
- def prepare_dataset(stage_1, opt_init_steps=300):
- """
- Prepares a text dataset for quantization.
- """
- dataset = load_dataset("laion/laion2B-en-aesthetic", streaming=True, split="train")
- train_dataset = dataset.shuffle(seed=RANDOM_SEED, buffer_size=1000).take(opt_init_steps)
- dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=1)
- calibration_data = prepare_calibration_data(dataloader, stage_1)
- return calibration_data
-
-.. code:: ipython3
-
- %%time
-
- generator = torch.manual_seed(RANDOM_SEED)
- opt_init_steps = 300
- selection_prob = 0.5
- prompts_number = np.ceil(opt_init_steps // (min(N_DIFFUSION_STEPS, UNET_2_STEPS) * selection_prob))
-
- stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant=checkpoint_variant, torch_dtype=model_dtype)
- encoded_prompts = prepare_dataset(stage_1, int(prompts_number))
-
-
-.. parsed-literal::
-
- safety_checker/model.safetensors not found
-
- A mixture of fp16 and non-fp16 filenames will be loaded.
- Loaded fp16 filenames:
- [text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
- Loaded non-fp16 filenames:
- [safety_checker/pytorch_model.bin, watermarker/diffusion_pytorch_model.bin
- If this behavior is not expected, please check your folder structure.
- Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
- ```
- pip install accelerate
- ```
- .
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
- /home/ea/work/ov_venv/lib/python3.8/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
- return torch._C._cuda_getDeviceCount() > 0
-
-
-
-.. parsed-literal::
-
- Loading checkpoint shards: 0%| | 0/2 [00:00= self.prob:
- self.data_cache.append(*args)
- return super().__call__(*args, **kwargs)
-
-.. code:: ipython3
-
- stage_1.unet = UnetFirstStage(UNET_I_IR_PATH, stage_1.unet.config, dtype=model_dtype, device=device.value)
- stage_1.set_progress_bar_config(disable=True)
-
- stage_1_data_cache = []
- stage_1.unet.unet_openvino = CompiledModelDecorator(stage_1.unet.unet_openvino, prob=selection_prob, data_cache=stage_1_data_cache)
-
- generator = torch.manual_seed(RANDOM_SEED)
- stage_2_inputs = [] # to speed up dataset preparation for stage 2 U-Net we can collect several images below
- for data in encoded_prompts:
- prompt_embeds, negative_embeds = data
- image = stage_1(
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=N_DIFFUSION_STEPS,
- ).images
- stage_2_inputs.append((image, prompt_embeds, negative_embeds))
-
- if len(stage_1_data_cache) >= opt_init_steps:
- break
-
-Quantize first stage U-Net
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- %%time
-
- ov_model = core.read_model(UNET_I_IR_PATH)
- stage_1_calibration_dataset = nncf.Dataset(stage_1_data_cache, lambda x: x)
-
- quantized_model = nncf.quantize(
- model=ov_model,
- calibration_dataset=stage_1_calibration_dataset,
- model_type=nncf.ModelType.TRANSFORMER,
- advanced_parameters=nncf.AdvancedQuantizationParameters(smooth_quant_alpha=0.25),
- )
-
- UNET_I_INT8_PATH = "_optimized.".join(UNET_I_IR_PATH.as_posix().split("."))
- ov.save_model(quantized_model, UNET_I_INT8_PATH)
-
-
-.. parsed-literal::
-
- Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [01:35<00:00, 3.14it/s]
- Applying Smooth Quant: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:04<00:00, 17.55it/s]
- Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [05:44<00:00, 1.15s/it]
- Applying Fast Bias correction: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 268/268 [00:35<00:00, 7.50it/s]
-
-
-.. parsed-literal::
-
- CPU times: user 1h 8min 46s, sys: 1min 22s, total: 1h 10min 8s
- Wall time: 9min 46s
-
-
-.. code:: ipython3
-
- %%time
-
- from tqdm.notebook import tqdm
-
- start = len(stage_2_inputs)
- for i, data in tqdm(enumerate(encoded_prompts[start:])):
- prompt_embeds, negative_embeds = data
- image = stage_1(
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=N_DIFFUSION_STEPS,
- ).images
- stage_2_inputs.append((image, prompt_embeds, negative_embeds))
-
-
-
-.. parsed-literal::
-
- 0it [00:00, ?it/s]
-
-
-.. parsed-literal::
-
- CPU times: user 1h 17min 46s, sys: 44.9 s, total: 1h 18min 31s
- Wall time: 4min 46s
-
-
-.. code:: ipython3
-
- %%time
-
- generator = torch.manual_seed(RANDOM_SEED)
- opt_init_steps = 300
-
- stage_2 = DiffusionPipeline.from_pretrained(
- "DeepFloyd/IF-II-M-v1.0",
- text_encoder=None,
- variant=checkpoint_variant,
- torch_dtype=model_dtype,
- )
- stage_2.set_progress_bar_config(disable=True)
-
- stage_2.unet = UnetSecondStage(UNET_II_IR_PATH, stage_2.unet.config, dtype=model_dtype, device=device.value)
- stage_2_data_cache = []
- stage_2.unet.unet_openvino = CompiledModelDecorator(stage_2.unet.unet_openvino, prob=selection_prob, data_cache=stage_2_data_cache)
-
- for data in tqdm(stage_2_inputs):
- image, prompt_embeds, negative_embeds = data
- image = stage_2(
- image=image,
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=UNET_2_STEPS,
- ).images
-
- if len(stage_2_data_cache) >= opt_init_steps:
- break
-
-
-.. parsed-literal::
-
-
- A mixture of fp16 and non-fp16 filenames will be loaded.
- Loaded fp16 filenames:
- [text_encoder/model.fp16-00001-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, safety_checker/model.fp16.safetensors]
- Loaded non-fp16 filenames:
- [watermarker/diffusion_pytorch_model.safetensors
- If this behavior is not expected, please check your folder structure.
- Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
- ```
- pip install accelerate
- ```
- .
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00`__.
-
-.. code:: ipython3
-
- prompt = "ultra close color photo portrait of rainbow owl with deer horns in the woods"
- negative_prompt = "blurred unreal uncentered occluded"
-
-.. code:: ipython3
-
- %%time
-
- stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-M-v1.0", variant=checkpoint_variant, torch_dtype=model_dtype)
-
- # Initialize the First Stage U-Net wrapper class
- stage_1.unet = UnetFirstStage(UNET_I_INT8_PATH, stage_1.unet.config, dtype=model_dtype, device=device.value)
-
- stage_1.text_encoder = TextEncoder(TEXT_ENCODER_INT8_IR_PATH, dtype=model_dtype, device=device.value)
- print("The model has been loaded")
-
- # Generate text embeddings
- prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt, negative_prompt=negative_prompt)
-
- # Fix PRNG seed
- generator = torch.manual_seed(RANDOM_SEED)
-
- # Inference
- image = stage_1(
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_embeds,
- generator=generator,
- output_type="pt",
- num_inference_steps=N_DIFFUSION_STEPS,
- ).images
-
- # Show the image
- pt_to_pil(image)[0]
-
-
-.. parsed-literal::
-
- safety_checker/model.safetensors not found
-
- A mixture of fp16 and non-fp16 filenames will be loaded.
- Loaded fp16 filenames:
- [text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
- Loaded non-fp16 filenames:
- [safety_checker/pytorch_model.bin, watermarker/diffusion_pytorch_model.bin
- If this behavior is not expected, please check your folder structure.
- Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:
- ```
- pip install accelerate
- ```
- .
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00`__.
-
- **NOTE**: For more accurate performance, run ``benchmark_app`` in a
- terminal/command prompt after closing other applications. Run
- ``benchmark_app --help`` to see an overview of all command-line
- options.
-
-.. code:: ipython3
-
- import re
-
-
- def get_fps(benchmark_output: str):
- parsed_output = [line for line in benchmark_output if "Throughput:" in line]
- fps = re.findall(r"\d+\.\d+", parsed_output[0])[0]
- return fps
-
-Text encoder
-
-.. code:: ipython3
-
- benchmark_output = !benchmark_app -m $TEXT_ENCODER_IR_PATH -d $device.value -api async
- original_fps = get_fps(benchmark_output)
- print(f"FP16 Text Encoder Throughput: {original_fps} FPS")
-
- benchmark_output = (
- !benchmark_app -m $TEXT_ENCODER_INT8_IR_PATH -d $device.value -api async
- )
- optimized_fps = get_fps(benchmark_output)
- print(f"INT8 Text Encoder Throughput: {optimized_fps} FPS")
- print(f"Text encoder speed up: {float(optimized_fps) / float(original_fps)}")
-
-
-.. parsed-literal::
-
- FP16 Text Encoder Throughput: 0.99 FPS
- INT8 Text Encoder Throughput: 2.47 FPS
- Text encoder speed up: 2.4949494949494953
-
-
-First stage UNet
-
-.. code:: ipython3
-
- benchmark_output = !benchmark_app -m $UNET_I_IR_PATH -d $device.value -api async
- original_fps = get_fps(benchmark_output)
- print(f"FP16 1 stage U-Net Throughput: {original_fps} FPS")
-
- benchmark_output = !benchmark_app -m $UNET_I_INT8_PATH -d $device.value -api async
- optimized_fps = get_fps(benchmark_output)
- print(f"INT8 1 stage U-Net Throughput: {optimized_fps} FPS")
- print(f"1 stage U-Net speed up: {float(optimized_fps) / float(original_fps)}")
-
-
-.. parsed-literal::
-
- FP16 1 stage U-Net Throughput: 4.65 FPS
- INT8 1 stage U-Net Throughput: 12.06 FPS
- 1 stage U-Net speed up: 2.593548387096774
-
-
-Second stage UNet
-
-.. code:: ipython3
-
- benchmark_output = !benchmark_app -m $UNET_II_IR_PATH -d $device.value -api async
- original_fps = get_fps(benchmark_output)
- print(f"FP16 2 stage U-Net Throughput: {original_fps} FPS")
-
- benchmark_output = !benchmark_app -m $UNET_II_INT8_PATH -d $device.value -api async
- optimized_fps = get_fps(benchmark_output)
- print(f"INT8 2 stage U-Net Throughput: {optimized_fps} FPS")
- print(f"2 stage U-Net speed up: {float(optimized_fps) / float(original_fps)}")
-
-
-.. parsed-literal::
-
- FP16 2 stage U-Net Throughput: 0.28 FPS
- INT8 2 stage U-Net Throughput: 0.92 FPS
- 2 stage U-Net speed up: 3.2857142857142856
-
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.jpg b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.jpg
deleted file mode 100644
index 6366c21515a..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:811dbf6cfc44f8eb6babf8771a3a98edb04aab2a92626add147b09205858dbae
-size 2577
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.png b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.png
deleted file mode 100644
index e3cbc9ce7c7..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_23_6.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8734c254db111a957ea26abd818829ae86fc04d0a007320bc978e4baf2d60913
-size 11369
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.jpg b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.jpg
deleted file mode 100644
index e4317f1326d..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2292f0150cb59cb15ecc6719ad2f260b3d366d768c1ced1d3b96327a029a776a
-size 23341
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.png b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.png
deleted file mode 100644
index 9d485b5b570..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_24_5.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d94733cc071e9db841a5b317e4a91919e608b238ca613df63d6bf26064ff9e0f
-size 164221
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.jpg b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.jpg
deleted file mode 100644
index 66547cbf23c..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:17806ca0d95899c5a82007a9719ef58a43a84ac1843beda3c4825a5d031ed57c
-size 210327
diff --git a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.png b/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.png
deleted file mode 100644
index de070b06159..00000000000
--- a/docs/notebooks/deep-floyd-if-optimize-with-output_files/deep-floyd-if-optimize-with-output_25_1.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:74a8d612863d32d45ed612993997e6c0d773e540b0f7eb619c543156d0231e29
-size 1937722
diff --git a/docs/notebooks/depth-anything-with-output.rst b/docs/notebooks/depth-anything-with-output.rst
index 482ff4c7735..1003219a81a 100644
--- a/docs/notebooks/depth-anything-with-output.rst
+++ b/docs/notebooks/depth-anything-with-output.rst
@@ -75,501 +75,17 @@ Prerequisites
.. parsed-literal::
Cloning into 'Depth-Anything'...
-
-
-.. parsed-literal::
-
remote: Enumerating objects: 421, done.[K
- remote: Counting objects: 0% (1/144)[K
-remote: Counting objects: 1% (2/144)[K
-remote: Counting objects: 2% (3/144)[K
-remote: Counting objects: 3% (5/144)[K
-remote: Counting objects: 4% (6/144)[K
-remote: Counting objects: 5% (8/144)[K
-remote: Counting objects: 6% (9/144)[K
-remote: Counting objects: 7% (11/144)[K
-remote: Counting objects: 8% (12/144)[K
-remote: Counting objects: 9% (13/144)[K
-remote: Counting objects: 10% (15/144)[K
-remote: Counting objects: 11% (16/144)[K
-remote: Counting objects: 12% (18/144)[K
-remote: Counting objects: 13% (19/144)[K
-remote: Counting objects: 14% (21/144)[K
-remote: Counting objects: 15% (22/144)[K
-remote: Counting objects: 16% (24/144)[K
-remote: Counting objects: 17% (25/144)[K
-remote: Counting objects: 18% (26/144)[K
-remote: Counting objects: 19% (28/144)[K
-remote: Counting objects: 20% (29/144)[K
-remote: Counting objects: 21% (31/144)[K
-remote: Counting objects: 22% (32/144)[K
-remote: Counting objects: 23% (34/144)[K
-remote: Counting objects: 24% (35/144)[K
-remote: Counting objects: 25% (36/144)[K
-remote: Counting objects: 26% (38/144)[K
-remote: Counting objects: 27% (39/144)[K
-remote: Counting objects: 28% (41/144)[K
-remote: Counting objects: 29% (42/144)[K
-remote: Counting objects: 30% (44/144)[K
-remote: Counting objects: 31% (45/144)[K
-remote: Counting objects: 32% (47/144)[K
-remote: Counting objects: 33% (48/144)[K
-remote: Counting objects: 34% (49/144)[K
-remote: Counting objects: 35% (51/144)[K
-remote: Counting objects: 36% (52/144)[K
-remote: Counting objects: 37% (54/144)[K
-remote: Counting objects: 38% (55/144)[K
-remote: Counting objects: 39% (57/144)[K
-remote: Counting objects: 40% (58/144)[K
-remote: Counting objects: 41% (60/144)[K
-remote: Counting objects: 42% (61/144)[K
-remote: Counting objects: 43% (62/144)[K
-remote: Counting objects: 44% (64/144)[K
-remote: Counting objects: 45% (65/144)[K
-remote: Counting objects: 46% (67/144)[K
-remote: Counting objects: 47% (68/144)[K
-remote: Counting objects: 48% (70/144)[K
-remote: Counting objects: 49% (71/144)[K
-remote: Counting objects: 50% (72/144)[K
-remote: Counting objects: 51% (74/144)[K
-remote: Counting objects: 52% (75/144)[K
-remote: Counting objects: 53% (77/144)[K
-remote: Counting objects: 54% (78/144)[K
-remote: Counting objects: 55% (80/144)[K
-remote: Counting objects: 56% (81/144)[K
-remote: Counting objects: 57% (83/144)[K
-remote: Counting objects: 58% (84/144)[K
-remote: Counting objects: 59% (85/144)[K
-remote: Counting objects: 60% (87/144)[K
-remote: Counting objects: 61% (88/144)[K
-remote: Counting objects: 62% (90/144)[K
-remote: Counting objects: 63% (91/144)[K
-remote: Counting objects: 64% (93/144)[K
-remote: Counting objects: 65% (94/144)[K
-remote: Counting objects: 66% (96/144)[K
-remote: Counting objects: 67% (97/144)[K
-remote: Counting objects: 68% (98/144)[K
-remote: Counting objects: 69% (100/144)[K
-remote: Counting objects: 70% (101/144)[K
-remote: Counting objects: 71% (103/144)[K
-remote: Counting objects: 72% (104/144)[K
-remote: Counting objects: 73% (106/144)[K
-remote: Counting objects: 74% (107/144)[K
-remote: Counting objects: 75% (108/144)[K
-remote: Counting objects: 76% (110/144)[K
-remote: Counting objects: 77% (111/144)[K
-remote: Counting objects: 78% (113/144)[K
-remote: Counting objects: 79% (114/144)[K
-remote: Counting objects: 80% (116/144)[K
-remote: Counting objects: 81% (117/144)[K
-remote: Counting objects: 82% (119/144)[K
-remote: Counting objects: 83% (120/144)[K
-remote: Counting objects: 84% (121/144)[K
-remote: Counting objects: 85% (123/144)[K
-remote: Counting objects: 86% (124/144)[K
-remote: Counting objects: 87% (126/144)[K
-remote: Counting objects: 88% (127/144)[K
-remote: Counting objects: 89% (129/144)[K
-remote: Counting objects: 90% (130/144)[K
-remote: Counting objects: 91% (132/144)[K
-remote: Counting objects: 92% (133/144)[K
-remote: Counting objects: 93% (134/144)[K
-remote: Counting objects: 94% (136/144)[K
-remote: Counting objects: 95% (137/144)[K
-remote: Counting objects: 96% (139/144)[K
-remote: Counting objects: 97% (140/144)[K
-remote: Counting objects: 98% (142/144)[K
-remote: Counting objects: 99% (143/144)[K
-remote: Counting objects: 100% (144/144)[K
-remote: Counting objects: 100% (144/144), done.[K
- remote: Compressing objects: 0% (1/105)[K
-remote: Compressing objects: 1% (2/105)[K
-remote: Compressing objects: 2% (3/105)[K
-remote: Compressing objects: 3% (4/105)[K
-remote: Compressing objects: 4% (5/105)[K
-remote: Compressing objects: 5% (6/105)[K
-remote: Compressing objects: 6% (7/105)[K
-remote: Compressing objects: 7% (8/105)[K
-remote: Compressing objects: 8% (9/105)[K
-remote: Compressing objects: 9% (10/105)[K
-remote: Compressing objects: 10% (11/105)[K
-remote: Compressing objects: 11% (12/105)[K
-remote: Compressing objects: 12% (13/105)[K
-remote: Compressing objects: 13% (14/105)[K
-remote: Compressing objects: 14% (15/105)[K
-remote: Compressing objects: 15% (16/105)[K
-remote: Compressing objects: 16% (17/105)[K
-remote: Compressing objects: 17% (18/105)[K
-remote: Compressing objects: 18% (19/105)[K
-remote: Compressing objects: 19% (20/105)[K
-remote: Compressing objects: 20% (21/105)[K
-remote: Compressing objects: 21% (23/105)[K
-remote: Compressing objects: 22% (24/105)[K
-remote: Compressing objects: 23% (25/105)[K
-remote: Compressing objects: 24% (26/105)[K
-remote: Compressing objects: 25% (27/105)[K
-remote: Compressing objects: 26% (28/105)[K
-remote: Compressing objects: 27% (29/105)[K
-remote: Compressing objects: 28% (30/105)[K
-remote: Compressing objects: 29% (31/105)[K
-remote: Compressing objects: 30% (32/105)[K
-remote: Compressing objects: 31% (33/105)[K
-remote: Compressing objects: 32% (34/105)[K
-remote: Compressing objects: 33% (35/105)[K
-remote: Compressing objects: 34% (36/105)[K
-remote: Compressing objects: 35% (37/105)[K
-remote: Compressing objects: 36% (38/105)[K
-remote: Compressing objects: 37% (39/105)[K
-remote: Compressing objects: 38% (40/105)[K
-remote: Compressing objects: 39% (41/105)[K
-remote: Compressing objects: 40% (42/105)[K
-remote: Compressing objects: 41% (44/105)[K
-remote: Compressing objects: 42% (45/105)[K
-remote: Compressing objects: 43% (46/105)[K
-remote: Compressing objects: 44% (47/105)[K
-remote: Compressing objects: 45% (48/105)[K
-remote: Compressing objects: 46% (49/105)[K
-remote: Compressing objects: 47% (50/105)[K
-remote: Compressing objects: 48% (51/105)[K
-remote: Compressing objects: 49% (52/105)[K
-remote: Compressing objects: 50% (53/105)[K
-remote: Compressing objects: 51% (54/105)[K
-remote: Compressing objects: 52% (55/105)[K
-remote: Compressing objects: 53% (56/105)[K
-remote: Compressing objects: 54% (57/105)[K
-remote: Compressing objects: 55% (58/105)[K
-remote: Compressing objects: 56% (59/105)[K
-remote: Compressing objects: 57% (60/105)[K
-remote: Compressing objects: 58% (61/105)[K
-remote: Compressing objects: 59% (62/105)[K
-remote: Compressing objects: 60% (63/105)[K
-remote: Compressing objects: 61% (65/105)[K
-remote: Compressing objects: 62% (66/105)[K
-remote: Compressing objects: 63% (67/105)[K
-remote: Compressing objects: 64% (68/105)[K
-remote: Compressing objects: 65% (69/105)[K
-remote: Compressing objects: 66% (70/105)[K
-remote: Compressing objects: 67% (71/105)[K
-remote: Compressing objects: 68% (72/105)[K
-remote: Compressing objects: 69% (73/105)[K
-remote: Compressing objects: 70% (74/105)[K
-remote: Compressing objects: 71% (75/105)[K
-remote: Compressing objects: 72% (76/105)[K
-remote: Compressing objects: 73% (77/105)[K
-remote: Compressing objects: 74% (78/105)[K
-remote: Compressing objects: 75% (79/105)[K
-remote: Compressing objects: 76% (80/105)[K
-remote: Compressing objects: 77% (81/105)[K
-remote: Compressing objects: 78% (82/105)[K
-remote: Compressing objects: 79% (83/105)[K
-remote: Compressing objects: 80% (84/105)[K
-remote: Compressing objects: 81% (86/105)[K
-remote: Compressing objects: 82% (87/105)[K
-remote: Compressing objects: 83% (88/105)[K
-remote: Compressing objects: 84% (89/105)[K
-remote: Compressing objects: 85% (90/105)[K
-remote: Compressing objects: 86% (91/105)[K
-remote: Compressing objects: 87% (92/105)[K
-remote: Compressing objects: 88% (93/105)[K
-remote: Compressing objects: 89% (94/105)[K
-remote: Compressing objects: 90% (95/105)[K
-remote: Compressing objects: 91% (96/105)[K
-remote: Compressing objects: 92% (97/105)[K
-remote: Compressing objects: 93% (98/105)[K
-remote: Compressing objects: 94% (99/105)[K
-remote: Compressing objects: 95% (100/105)[K
-remote: Compressing objects: 96% (101/105)[K
-remote: Compressing objects: 97% (102/105)[K
-remote: Compressing objects: 98% (103/105)[K
-remote: Compressing objects: 99% (104/105)[K
-remote: Compressing objects: 100% (105/105)[K
-remote: Compressing objects: 100% (105/105), done.[K
-
-
-.. parsed-literal::
-
- Receiving objects: 0% (1/421)
-
-.. parsed-literal::
-
- Receiving objects: 1% (5/421), 10.76 MiB | 21.55 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 1% (8/421), 19.25 MiB | 19.12 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 2% (9/421), 19.25 MiB | 19.12 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 3% (13/421), 31.03 MiB | 20.47 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 4% (17/421), 31.03 MiB | 20.47 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 4% (19/421), 43.39 MiB | 21.41 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 5% (22/421), 56.34 MiB | 22.23 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 5% (25/421), 56.34 MiB | 22.23 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 6% (26/421), 56.34 MiB | 22.23 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 7% (30/421), 69.25 MiB | 22.74 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 8% (34/421), 69.25 MiB | 22.74 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 8% (37/421), 82.28 MiB | 22.91 MiB/s
-Receiving objects: 9% (38/421), 82.28 MiB | 22.91 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 10% (43/421), 110.97 MiB | 24.17 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 11% (47/421), 110.97 MiB | 24.17 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 11% (49/421), 124.93 MiB | 24.86 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 12% (51/421), 124.93 MiB | 24.86 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 13% (55/421), 124.93 MiB | 24.86 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 14% (59/421), 139.53 MiB | 26.18 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 14% (60/421), 152.32 MiB | 26.35 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 15% (64/421), 167.60 MiB | 26.99 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 15% (66/421), 167.60 MiB | 26.99 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 16% (68/421), 183.43 MiB | 27.60 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 17% (72/421), 183.43 MiB | 27.60 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 18% (76/421), 183.43 MiB | 27.60 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 19% (80/421), 200.00 MiB | 28.39 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 20% (85/421), 200.00 MiB | 28.39 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 20% (88/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 21% (89/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 22% (93/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 23% (97/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 24% (102/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 25% (106/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 26% (110/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 27% (114/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 28% (118/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 29% (123/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 30% (127/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 31% (131/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 32% (135/421), 216.59 MiB | 29.40 MiB/s
-Receiving objects: 33% (139/421), 216.59 MiB | 29.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 34% (144/421), 233.69 MiB | 29.70 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 35% (148/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 36% (152/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 37% (156/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 38% (160/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 39% (165/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 40% (169/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 41% (173/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 42% (177/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 43% (182/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 44% (186/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 45% (190/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 46% (194/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 47% (198/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 48% (203/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 49% (207/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 50% (211/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 51% (215/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 52% (219/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 53% (224/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 54% (228/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 55% (232/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 56% (236/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 57% (240/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 58% (245/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 59% (249/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 60% (253/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 61% (257/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 62% (262/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 63% (266/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 64% (270/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 65% (274/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 66% (278/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 67% (283/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 68% (287/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 69% (291/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 70% (295/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 71% (299/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 72% (304/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 73% (308/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 74% (312/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 75% (316/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 76% (320/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 77% (325/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 78% (329/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 79% (333/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 80% (337/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 81% (342/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 82% (346/421), 233.69 MiB | 29.70 MiB/s
-remote: Total 421 (delta 101), reused 40 (delta 39), pack-reused 277[K
- Receiving objects: 83% (350/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 84% (354/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 85% (358/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 86% (363/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 87% (367/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 88% (371/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 89% (375/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 90% (379/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 91% (384/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 92% (388/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 93% (392/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 94% (396/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 95% (400/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 96% (405/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 97% (409/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 98% (413/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 99% (417/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 100% (421/421), 233.69 MiB | 29.70 MiB/s
-Receiving objects: 100% (421/421), 237.89 MiB | 27.05 MiB/s, done.
- Resolving deltas: 0% (0/144)
-Resolving deltas: 1% (2/144)
-Resolving deltas: 2% (3/144)
-Resolving deltas: 4% (6/144)
-Resolving deltas: 5% (8/144)
-Resolving deltas: 6% (10/144)
-Resolving deltas: 7% (11/144)
-Resolving deltas: 8% (12/144)
-Resolving deltas: 9% (14/144)
-Resolving deltas: 13% (19/144)
-Resolving deltas: 16% (24/144)
-Resolving deltas: 21% (31/144)
-Resolving deltas: 25% (36/144)
-Resolving deltas: 27% (39/144)
-Resolving deltas: 28% (41/144)
-Resolving deltas: 29% (42/144)
-Resolving deltas: 40% (58/144)
-Resolving deltas: 41% (60/144)
-Resolving deltas: 43% (63/144)
-Resolving deltas: 46% (67/144)
-Resolving deltas: 49% (71/144)
-Resolving deltas: 50% (72/144)
-Resolving deltas: 65% (94/144)
-Resolving deltas: 66% (96/144)
-Resolving deltas: 70% (102/144)
-Resolving deltas: 71% (103/144)
-Resolving deltas: 72% (104/144)
-Resolving deltas: 74% (107/144)
-Resolving deltas: 77% (111/144)
-Resolving deltas: 78% (113/144)
-Resolving deltas: 81% (118/144)
-Resolving deltas: 83% (120/144)
-Resolving deltas: 84% (121/144)
-Resolving deltas: 100% (144/144)
-Resolving deltas: 100% (144/144), done.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything
-
-
-.. parsed-literal::
-
+ remote: Counting objects: 100% (144/144), done.[K
+ remote: Compressing objects: 100% (105/105), done.[K
+ remote: Total 421 (delta 101), reused 43 (delta 39), pack-reused 277[K
+ Receiving objects: 100% (421/421), 237.89 MiB | 26.48 MiB/s, done.
+ Resolving deltas: 100% (144/144), done.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything
+ Note: you may need to restart the kernel to use updated packages.
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -617,10 +133,6 @@ DepthAnything family.
.. parsed-literal::
xFormers not available
-
-
-.. parsed-literal::
-
xFormers not available
@@ -761,17 +273,13 @@ loading on device using ``core.complie_model``.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert H % patch_H == 0, f"Input image height {H} is not a multiple of patch height {patch_H}"
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:74: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/dinov2/layers/patch_embed.py:74: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert W % patch_W == 0, f"Input image width {W} is not a multiple of patch width: {patch_W}"
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/vision_transformer.py:183: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/torchhub/facebookresearch_dinov2_main/vision_transformer.py:183: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if npatch == N and w == h:
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/depth_anything/dpt.py:133: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/depth_anything/dpt.py:133: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
out = F.interpolate(out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True)
@@ -1063,7 +571,7 @@ Run inference on video
.. parsed-literal::
- Processed 60 frames in 13.27 seconds. Total FPS (including video processing): 4.52.Inference FPS: 10.65
+ Processed 60 frames in 13.33 seconds. Total FPS (including video processing): 4.50.Inference FPS: 10.52
Video saved to 'output/Coco Walking in Berkeley_depth_anything.mp4'.
@@ -1090,7 +598,7 @@ Run inference on video
.. parsed-literal::
Showing video saved at
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
If you cannot see the video in your browser, please click on the following link to download the video
@@ -1227,14 +735,10 @@ quantization code below may take some time.
.. parsed-literal::
- 2024-04-17 23:32:53.792726: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:32:53.826550: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-06 23:52:17.005825: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-06 23:52:17.039208: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:32:54.388126: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-06 23:52:17.604619: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@@ -1282,10 +786,6 @@ quantization code below may take some time.
.. parsed-literal::
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:48 ignored nodes were found by name in the NNCFGraph
@@ -1310,12 +810,6 @@ quantization code below may take some time.
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
- return Tensor(self.data * unwrap_tensor_data(other))
-
-
.. parsed-literal::
@@ -1408,10 +902,10 @@ data.
.. parsed-literal::
- Processed 60 frames in 12.69 seconds. Total FPS (including video processing): 4.73.Inference FPS: 12.74
+ Processed 60 frames in 12.84 seconds. Total FPS (including video processing): 4.67.Inference FPS: 12.79
Video saved to 'output/Coco Walking in Berkeley_depth_anything_int8.mp4'.
Showing video saved at
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/depth-anything/Depth-Anything/output/Coco Walking in Berkeley_depth_anything.mp4
If you cannot see the video in your browser, please click on the following link to download the video
@@ -1450,8 +944,8 @@ Compare model file size
.. parsed-literal::
FP16 model size: 47.11 MB
- INT8 model size: 24.27 MB
- Model compression rate: 1.942
+ INT8 model size: 24.41 MB
+ Model compression rate: 1.930
Compare inference time of the FP16 and INT8 models
@@ -1491,13 +985,9 @@ Tool
-
-.. raw:: html
-
-
-
-The overall methodology. Diagram taken from the VI-Depth repository.
-
-.. raw:: html
-
-
-
-.. raw:: html
-
-
-
-A visual-inertial depth estimation pipeline that integrates monocular
-depth estimation and visual-inertial odometry to produce dense depth
-estimates with metric scale has been presented by the authors. The
-approach consists of three stages:
-
-1. input processing, where RGB and inertial measurement unit (IMU) data
- feed into monocular depth estimation alongside visual-inertial
- odometry,
-2. global scale and shift alignment, where monocular depth estimates are
- fitted to sparse depth from visual inertial odometry (VIO) in a
- least-squares manner and
-3. learning-based dense scale alignment, where globally-aligned depth is
- locally realigned using a dense scale map regressed by the
- ScaleMapLearner (SML).
-
-The images at the bottom in the diagram above illustrate a Visual
-Odometry with Inertial and Depth (VOID) sample being processed through
-our pipeline; from left to right: the input RGB, ground truth depth,
-sparse depth from VIO, globally-aligned depth, scale map scaffolding,
-dense scale map regressed by SML, final depth output.
-
-.. raw:: html
-
-
-
-.. raw:: html
-
-
-
-An illustration of VOID samples being processed by the image pipeline.
-Image taken from the VI-Depth repository.
-
-.. raw:: html
-
-
-
-.. raw:: html
-
-
-
-We will be consulting the `VI-Depth
-repository `__ for the
-pre-processing, model transformations and basic utility code. A part of
-it has already been kept as it is in the `utils `__ directory. At
-the same time we will learn how to perform `model
-conversion `__
-for converting a model in a different format to the standard OpenVINO™
-IR model representation *via* another format.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Imports <#imports>`__
-- `Loading models and checkpoints <#loading-models-and-checkpoints>`__
-
- - `Cleaning up the model
- directory <#cleaning-up-the-model-directory>`__
- - `Transformation of models <#transformation-of-models>`__
-
- - `Dummy input creation <#dummy-input-creation>`__
- - `Conversion of depth model to OpenVINO IR
- format <#conversion-of-depth-model-to-openvino-ir-format>`__
-
- - `Select inference device <#select-inference-device>`__
- - `Compilation of depth model <#compilation-of-depth-model>`__
- - `Computation of scale and shift
- parameters <#computation-of-scale-and-shift-parameters>`__
-
- - `Conversion of Scale Map Learner model to OpenVINO IR
- format <#conversion-of-scale-map-learner-model-to-openvino-ir-format>`__
-
- - `Select inference device <#select-inference-device>`__
- - `Compilation of the ScaleMapLearner(SML)
- model <#compilation-of-the-scalemaplearnersml-model>`__
-
- - `Storing and visualizing dummy results
- obtained <#storing-and-visualizing-dummy-results-obtained>`__
-
- - `Running inference on a test
- image <#running-inference-on-a-test-image>`__
- - `Store and visualize Inference
- results <#store-and-visualize-inference-results>`__
-
- - `Cleaning up the data
- directory <#cleaning-up-the-data-directory>`__
-
- - `Concluding notes <#concluding-notes>`__
-
-Imports
-~~~~~~~
-
-
-
-.. code:: ipython3
-
- import platform
-
- # Download the correct version of the PyTorch deep learning library associated with image models
- # alongside the lightning module
- %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "openvino>=2024.0.0" torch torchvision "pytorch-lightning" "timm>=0.6.12" tqdm
-
- if platform.system() != "Windows":
- %pip install -q "matplotlib>=3.4"
- else:
- %pip install -q "matplotlib>=3.4,<3.7"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- import sys
- import matplotlib.pyplot as plt
- import matplotlib.image as mpimg
- import numpy as np
- import openvino as ov
- import torch
- import torchvision
- from pathlib import Path
- from shutil import rmtree
- from typing import Optional, Tuple
-
- # Fetch `notebook_utils` module
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
-
- open("notebook_utils.py", "w").write(r.text)
- from notebook_utils import download_file
-
- sys.path.append("vi_depth_utils")
- import data_loader
- import modules.midas.transforms as transforms
- import modules.midas.utils as utils
- from modules.estimator import LeastSquaresEstimator
- from modules.interpolator import Interpolator2D
- from modules.midas.midas_net_custom import MidasNet_small_videpth
-
-.. code:: ipython3
-
- # Ability to display images inline
- %matplotlib inline
-
-Loading models and checkpoints
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-The complete pipeline here requires only two models: one for depth
-estimation and a ScaleMapLearner model which is responsible for
-regressing a dense scale map. The table of models which has been given
-in the original `VI-Depth repo `__
-has been presented as it is for the users to download from.
-`VOID `__ is the name of the
-original dataset from on which these models have been trained. The
-numbers after the word **VOID** represent the checkpoint in the model
-obtained after training samples for sparse dense maps corresponding to
-:math:`150`, :math:`500` and :math:`1500` levels in the density map.
-Just *right-click* on any of the highlighted links and click on “Copy
-link address”. We shall use this link in the next cell to download the
-ScaleMapLearner model. *Interestingly*, the ScaleMapLearner decides the
-depth prediction model as you will see.
-
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| Depth Predictor | SML on VOID 150 | SML on VOID 500 | SML on VOID 1500 |
-+==================+=================================================================================================================================+=================================================================================================================================+==================================================================================================================================+
-| DPT-BEiT-Large | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| DPT-SwinV2-Large | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| DPT-Large | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| DPT-Hybrid | `model `__\ \* | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| DPT-SwinV2-Tiny | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| DPT-LeViT | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-| MiDaS-small | `model `__ | `model `__ | `model `__ |
-+------------------+---------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+
-
-\*Also available with pre-training on TartanAir:
-`model `__
-
-.. code:: ipython3
-
- # Base directory in which models would be stored as a pathlib.Path variable
- MODEL_DIR = Path("model")
-
- # Mapping between depth predictors and the corresponding scale map learners
- PREDICTOR_MODEL_MAP = {
- "dpt_beit_large_512": "DPT_BEiT_L_512",
- "dpt_swin2_large_384": "DPT_SwinV2_L_384",
- "dpt_large": "DPT_Large",
- "dpt_hybrid": "DPT_Hybrid",
- "dpt_swin2_tiny_256": "DPT_SwinV2_T_256",
- "dpt_levit_224": "DPT_LeViT_224",
- "midas_small": "MiDaS_small",
- }
-
-.. code:: ipython3
-
- # Create the model directory adjacent to the notebook and suppress errors if the directory already exists
- MODEL_DIR.mkdir(exist_ok=True)
-
- # Here we will be downloading the SML model corresponding to the MiDaS-small depth predictor for
- # the checkpoint captured after training on 1500 points of the density level. Suppress errors if the file already exists
- download_file(
- "https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.midas_small.nsamples.1500.ckpt",
- directory=MODEL_DIR,
- silent=True,
- )
-
- # Take a note of the samples. It would be of major use later on
- NSAMPLES = 1500
-
-
-
-.. parsed-literal::
-
- model/sml_model.dpredictor.midas_small.nsamples.1500.ckpt: 0%| | 0.00/208M [00:00 str:
- """
- Download a model from the pre-validated 'isl-org/MiDaS:2.1' set of releases on the GitHub repo
- while simultaneously trusting the repo permanently
-
- :param: depth_predictor: Any depth estimation model amongst the ones given at https://github.com/isl-org/VI-Depth#setup
- :param: remote_repo: The remote GitHub repo from where the models will be downloaded
- :returns: A PyTorch model callable
- """
-
- # Workaround for avoiding rate limit errors
- torch.hub._validate_not_a_forked_repo = lambda a, b, c: True
-
- return torch.hub.load(
- remote_repo,
- PREDICTOR_MODEL_MAP[depth_predictor],
- skip_validation=True,
- trust_repo=True,
- )
-
-.. code:: ipython3
-
- # Execute the above function so as to download the MiDaS-small model
- # and get the output of the model callable in return
- depth_model = get_model_for_predictor("midas_small")
-
-
-.. parsed-literal::
-
- Downloading: "https://github.com/intel-isl/MiDaS/zipball/master" to model/master.zip
-
-
-.. parsed-literal::
-
- Loading weights: None
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/hub.py:294: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour
- warnings.warn(
- Downloading: "https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master" to model/master.zip
-
-
-.. parsed-literal::
-
- Downloading: "https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_lite3-b733e338.pth" to model/checkpoints/tf_efficientnet_lite3-b733e338.pth
-
-
-.. parsed-literal::
-
- Downloading: "https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_small_256.pt" to model/checkpoints/midas_v21_small_256.pt
-
-
-.. parsed-literal::
-
-
- 0%| | 0.00/81.8M [00:00`__
-downloads a lot of unnecessary files. We shall move remove the
-unnecessary directories and files which were created during the download
-process.
-
-.. code:: ipython3
-
- # Remove unnecessary directories and files and suppress errors(if any)
- rmtree(path=str(MODEL_DIR / "intel-isl_MiDaS_master"), ignore_errors=True)
- rmtree(
- path=str(MODEL_DIR / "rwightman_gen-efficientnet-pytorch_master"),
- ignore_errors=True,
- )
- rmtree(path=str(MODEL_DIR / "checkpoints"), ignore_errors=True)
-
- # Check for the existence of the trusted list file and then remove
- list_file = Path(MODEL_DIR / "trusted_list")
- if list_file.is_file():
- list_file.unlink()
-
-Transformation of models
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Each of the models need an appropriate transformation which can be
-invoked by the ``get_model_transforms`` function. It needs only the
-``depth_predictor`` parameter and ``NSAMPLES`` defined above to work.
-The reason being that the ``ScaleMapLearner`` and the depth estimation
-model are always in direct correspondence with each other.
-
-.. code:: ipython3
-
- # Define important custom types
- type_transform_compose = torchvision.transforms.transforms.Compose
- type_compiled_model = ov.CompiledModel
-
-.. code:: ipython3
-
- def get_model_transforms(depth_predictor: str, nsamples: int) -> Tuple[type_transform_compose, type_transform_compose]:
- """
- Construct the transformation of the depth prediction model and the
- associated ScaleMapLearner model
-
- :param: depth_predictor: Any depth estimation model amongst the ones given at https://github.com/isl-org/VI-Depth#setup
- :param: nsamples: The no. of density levels for the depth map
- :returns: The transformed models as the resut of torchvision.transforms.Compose operations
- """
- model_transforms = transforms.get_transforms(depth_predictor, "void", str(nsamples))
- return model_transforms["depth_model"], model_transforms["sml_model"]
-
-.. code:: ipython3
-
- # Obtain transforms of both the models here
- depth_model_transform, scale_map_learner_transform = get_model_transforms(depth_predictor="midas_small", nsamples=NSAMPLES)
-
-Dummy input creation
-^^^^^^^^^^^^^^^^^^^^
-
-
-
-Dummy inputs help during conversion. Although ``ov.convert_model``
-accepts any dummy input for a single pass through the model and thereby
-enabling model conversion, the pre-processing required for the actual
-inputs later at inference using compiled models would be substantial. So
-we have decided that even dummy inputs should go through the proper
-transformation process so that the reader gets the idea of a
-*transformed image* being compiled by a *transformed model*.
-
-Also note down the width and height of the image which would be used
-multiple times later. Do note that this is constant throughout the
-dataset
-
-.. code:: ipython3
-
- IMAGE_H, IMAGE_W = 480, 640
-
- # Although you can always verify the same by uncommenting and running
- # the following lines
- # img = cv2.imread('data/image/dummy_img.png')
- # print(img.shape)
-
-.. code:: ipython3
-
- # Base directory in which data would be stored as a pathlib.Path variable
- DATA_DIR = Path("data")
-
- # Create the data directory tree adjacent to the notebook and suppress errors if the directory already exists
- # Create a directory each for the images and their corresponding depth maps
- DATA_DIR.mkdir(exist_ok=True)
- Path(DATA_DIR / "image").mkdir(exist_ok=True)
- Path(DATA_DIR / "sparse_depth").mkdir(exist_ok=True)
-
- # Download the dummy image and its depth scale (take a note of the image hashes for possible later use)
- # On the fly download is being done to avoid unnecessary memory/data load during testing and
- # creation of PRs
- download_file(
- "https://user-images.githubusercontent.com/22426058/254174385-161b9f0e-5991-4308-ba89-d81bc02bcb7c.png",
- filename="dummy_img.png",
- directory=Path(DATA_DIR / "image"),
- silent=True,
- )
- download_file(
- "https://user-images.githubusercontent.com/22426058/254174398-8c71c59f-0adf-43c6-ad13-c04431e02349.png",
- filename="dummy_depth.png",
- directory=Path(DATA_DIR / "sparse_depth"),
- silent=True,
- )
-
- # Load the dummy image and its depth scale
- dummy_input = data_loader.load_input_image("data/image/dummy_img.png")
- dummy_depth = data_loader.load_sparse_depth("data/sparse_depth/dummy_depth.png")
-
-
-
-.. parsed-literal::
-
- data/image/dummy_img.png: 0%| | 0.00/328k [00:00 torch.Tensor:
- """
- Transform the input_image for processing by a PyTorch depth estimation model
-
- :param: input_image: The input image obtained as a result of data_loader.load_input_image
- :param: depth_model_transform: The transformed depth model
- :param: device: The device on which the image would be allocated
- :returns: The transformed image suitable to be used as an input to the depth estimation model
- """
- input_height, input_width = np.shape(input_image)[:2]
-
- sample = {"image": input_image}
- sample = depth_model_transform(sample)
- im = sample["image"].to(device)
- return im.unsqueeze(0)
-
-.. code:: ipython3
-
- # Transform the dummy input image for the depth model
- transformed_dummy_image = transform_image_for_depth(input_image=dummy_input, depth_model_transform=depth_model_transform)
-
-Conversion of depth model to OpenVINO IR format
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-Starting from 2023.0.0 release, OpenVINO supports PyTorch model via
-conversion to OpenVINO Intermediate Representation format (IR). To have
-a depth estimation model in the OpenVINO™ IR format and then compile it,
-we shall follow the following steps:
-
-1. Use the ``depth_model`` callable to our advantage from the *Loading
- models and checkpoints* stage.
-2. Convert PyTorch model to OpenVINO model using OpenVINO Model
- conversion API and the transformed dummy input created earlier.
-3. Use the ``ov.save_model`` function from OpenVINO to serialize
- OpenVINO ``.xml`` and ``.bin`` files for next compilation skipping
- conversion step Alternatively serialization procedure may be avoided
- and compiled model may be obtained by directly using OpenVINO’s
- ``compile_model`` function.
-
-.. code:: ipython3
-
- # Evaluate the model to switch some operations from training mode to inference.
- depth_model.eval()
-
-
- # Check PyTorch model work with dummy input
- _ = depth_model(transformed_dummy_image)
-
- # convert model to OpenVINO IR
- ov_model = ov.convert_model(depth_model, example_input=(transformed_dummy_image,))
-
- # save model for next usage
- ov.save_model(ov_model, "depth_model.xml")
-
-Select inference device
-'''''''''''''''''''''''
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- core = ov.Core()
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-Compilation of depth model
-''''''''''''''''''''''''''
-
-
-
-Now we can go ahead and compile our depth model.
-
-.. code:: ipython3
-
- # Initialize OpenVINO Runtime.
- compiled_depth_model = core.compile_model(model=ov_model, device_name=device.value)
-
-.. code:: ipython3
-
- def run_depth_model(
- input_image_h: int,
- input_image_w: int,
- transformed_image: torch.Tensor,
- compiled_depth_model: type_compiled_model,
- ) -> np.ndarray:
- """
- Run the compiled_depth_model on the transformed_image of dimensions
- input_image_w x input_image_h
-
- :param: input_image_h: The height of the input image
- :param: input_image_w: The width of the input image
- :param: transformed_image: The transformed image suitable to be used as an input to the depth estimation model
- :returns:
- depth_pred: The depth prediction on the image as an np.ndarray type
-
- """
-
- # Obtain the last output layer separately
- output_layer_depth_model = compiled_depth_model.output(0)
-
- with torch.no_grad():
- # Perform computation like a standard OpenVINO compiled model
- depth_pred = torch.from_numpy(compiled_depth_model([transformed_image])[output_layer_depth_model])
- depth_pred = (
- torch.nn.functional.interpolate(
- depth_pred.unsqueeze(1),
- size=(input_image_h, input_image_w),
- mode="bicubic",
- align_corners=False,
- )
- .squeeze()
- .cpu()
- .numpy()
- )
-
- return depth_pred
-
-.. code:: ipython3
-
- # Run the compiled depth model using the dummy input
- # It will be used to compute the metrics associated with the ScaleMapLearner model
- # and hence obtain a compiled version of the same later
- depth_pred_dummy = run_depth_model(
- input_image_h=IMAGE_H,
- input_image_w=IMAGE_W,
- transformed_image=transformed_dummy_image,
- compiled_depth_model=compiled_depth_model,
- )
-
-Computation of scale and shift parameters
-'''''''''''''''''''''''''''''''''''''''''
-
-
-
-Computation of these parameters required the depth estimation model
-output from the previous step. These are the regression based parameters
-the ScaleMapLearner model deals with. An utility function for the
-purpose has already been created.
-
-.. code:: ipython3
-
- def compute_global_scale_and_shift(
- input_sparse_depth: np.ndarray,
- validity_map: Optional[np.ndarray],
- depth_pred: np.ndarray,
- min_pred: float = 0.1,
- max_pred: float = 8.0,
- min_depth: float = 0.2,
- max_depth: float = 5.0,
- ) -> Tuple[np.ndarray, np.ndarray]:
- """
- Compute the global scale and shift alignment required for SML model to work on
- with the input_sparse_depth map being provided and the depth estimation output depth_pred
- being provided with an optional validity_map
-
- :param: input_sparse_depth: The depth map of the input image
- :param: validity_map: An optional depth map associated with the original input image
- :param: depth_pred: The depth estimate obtained after running the depth model on the input image
- :param: min_pred: Lower bound for predicted depth values
- :param: max_pred: Upper bound for predicted depth values
- :param: min_depth: Min valid depth when evaluating
- :param: max_depth: Max valid depth when evaluating
- :returns:
- int_depth: The depth estimate for the SML regression model
- int_scales: The scale to be used for the SML regression model
-
- """
-
- input_sparse_depth_valid = (input_sparse_depth < max_depth) * (input_sparse_depth > min_depth)
- if validity_map is not None:
- input_sparse_depth_valid *= validity_map.astype(np.bool)
-
- input_sparse_depth_valid = input_sparse_depth_valid.astype(bool)
- input_sparse_depth[~input_sparse_depth_valid] = np.inf # set invalid depth
- input_sparse_depth = 1.0 / input_sparse_depth
-
- # global scale and shift alignment
- GlobalAlignment = LeastSquaresEstimator(estimate=depth_pred, target=input_sparse_depth, valid=input_sparse_depth_valid)
- GlobalAlignment.compute_scale_and_shift()
- GlobalAlignment.apply_scale_and_shift()
- GlobalAlignment.clamp_min_max(clamp_min=min_pred, clamp_max=max_pred)
- int_depth = GlobalAlignment.output.astype(np.float32)
-
- # interpolation of scale map
- assert np.sum(input_sparse_depth_valid) >= 3, "not enough valid sparse points"
- ScaleMapInterpolator = Interpolator2D(
- pred_inv=int_depth,
- sparse_depth_inv=input_sparse_depth,
- valid=input_sparse_depth_valid,
- )
- ScaleMapInterpolator.generate_interpolated_scale_map(interpolate_method="linear", fill_corners=False)
-
- int_scales = ScaleMapInterpolator.interpolated_scale_map.astype(np.float32)
- int_scales = utils.normalize_unit_range(int_scales)
-
- return int_depth, int_scales
-
-.. code:: ipython3
-
- # Call the function on the dummy depth map we loaded in the dummy_depth variable
- # with all default settings and store in appropriate variables
- d_depth, d_scales = compute_global_scale_and_shift(input_sparse_depth=dummy_depth, validity_map=None, depth_pred=depth_pred_dummy)
-
-.. code:: ipython3
-
- def transform_image_for_depth_scale(
- input_image: np.ndarray,
- scale_map_learner_transform: type_transform_compose,
- int_depth: np.ndarray,
- int_scales: np.ndarray,
- device: torch.device = "cpu",
- ) -> Tuple[torch.Tensor, torch.Tensor]:
- """
- Transform the input_image for processing by a PyTorch SML model
-
- :param: input_image: The input image obtained as a result of data_loader.load_input_image
- :param: scale_map_learner_transform: The transformed SML model
- :param: int_depth: The depth estimate for the SML regression model
- :param: int_scales: he scale to be used for the SML regression model
- :param: device: The device on which the image would be allocated
- :returns: The transformed tensor inputs suitable to be used with an SML model
- """
-
- sample = {
- "image": input_image,
- "int_depth": int_depth,
- "int_scales": int_scales,
- "int_depth_no_tf": int_depth,
- }
- sample = scale_map_learner_transform(sample)
- x = torch.cat([sample["int_depth"], sample["int_scales"]], 0)
- x = x.to(device)
- d = sample["int_depth_no_tf"].to(device)
-
- return x.unsqueeze(0), d.unsqueeze(0)
-
-.. code:: ipython3
-
- # Transform the dummy input image for the ScaleMapLearner model
- # Note that this will lead to a tuple as an output. Both the elements
- # which is fed to ScaleMapLearner during the conversion process to onxx
- transformed_dummy_image_scale = transform_image_for_depth_scale(
- input_image=dummy_input,
- scale_map_learner_transform=scale_map_learner_transform,
- int_depth=d_depth,
- int_scales=d_scales,
- )
-
-Conversion of Scale Map Learner model to OpenVINO IR format
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-The OpenVINO™ toolkit provides direct method of converting PyTorch
-models to the intermediate representation format. To have the associated
-ScaleMapLearner in the OpenVINO™ IR format and then compile it, we shall
-follow the following steps:
-
-1. Load the model in memory via instantiating the
- ``modules.midas.midas_net_custom.MidasNet_small_videpth`` class and
- passing the downloaded checkpoint earlier as an argument.
-2. Convert PyTorch model to OpenVINO model using OpenVINO Model
- conversion API and the transformed dummy input created earlier.
-3. Use the ``ov.save_model`` function from OpenVINO to serialize
- OpenVINO ``.xml`` and ``.bin`` files for next compilation skipping
- conversion step Alternatively serialization procedure may be avoided
- and compiled model may be obtained by directly using OpenVINO’s
- ``compile_model`` function.
-
-If the name of the ``.ckpt`` file is too much to handle, here is the
-common format of all checkpoint files from the model releases.
-
- - sml_model.dpredictor..nsamples..ckpt
- - Replace and with the depth estimation
- model name and the no. of levels of depth density the SML model
- has been trained on
- - E.g. sml_model.dpredictor.dpt_hybrid.nsamples.500.ckpt will be the
- file name corresponding to the SML model based on the dpt_hybrid
- depth predictor and has been trained on 500 points of the density
- level on the depth map
-
-.. code:: ipython3
-
- # Run with the same min_pred and max_pred arguments which were used to compute
- # global scale and shift alignment
- scale_map_learner = MidasNet_small_videpth(
- path=str(MODEL_DIR / "sml_model.dpredictor.midas_small.nsamples.1500.ckpt"),
- min_pred=0.1,
- max_pred=8.0,
- )
-
-
-.. parsed-literal::
-
- Loading weights: model/sml_model.dpredictor.midas_small.nsamples.1500.ckpt
-
-
-.. parsed-literal::
-
- Downloading: "https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master" to model/master.zip
-
-
-.. code:: ipython3
-
- # As usual, since the MidasNet_small_videpthc class internally downloads a repo again from torch hub
- # we shall clean the same since the model callable is now available to us
- # Remove unnecessary directories and files and suppress errors(if any)
- rmtree(
- path=str(MODEL_DIR / "rwightman_gen-efficientnet-pytorch_master"),
- ignore_errors=True,
- )
-
- # Check for the existence of the trusted list file and then remove
- list_file = Path(MODEL_DIR / "trusted_list")
- if list_file.is_file():
- list_file.unlink()
-
-.. code:: ipython3
-
- # Evaluate the model to switch some operations from training mode to inference.
- scale_map_learner.eval()
-
- # Store the tuple of dummy variables into separate variables for easier reference
- x_dummy, d_dummy = transformed_dummy_image_scale
-
- # Check that PyTorch model works with dummy input
- _ = scale_map_learner(x_dummy, d_dummy)
-
- # Convert model to OpenVINO IR
- scale_map_learner = ov.convert_model(scale_map_learner, example_input=(x_dummy, d_dummy))
-
- # Save model on disk for next usage
- ov.save_model(scale_map_learner, "scale_map_learner.xml")
-
-Select inference device
-'''''''''''''''''''''''
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-Compilation of the ScaleMapLearner(SML) model
-'''''''''''''''''''''''''''''''''''''''''''''
-
-
-
-Now we can go ahead and compile our SML model.
-
-.. code:: ipython3
-
- # In the situation where you are unaware of the correct device to compile your
- # model in, just set device_name='AUTO' and let OpenVINO decide for you
- compiled_scale_map_learner = core.compile_model(model=scale_map_learner, device_name=device.value)
-
-.. code:: ipython3
-
- def run_depth_scale_model(
- input_image_h: int,
- input_image_w: int,
- transformed_image_for_depth_scale: Tuple[torch.Tensor, torch.Tensor],
- compiled_scale_map_learner: type_compiled_model,
- ) -> np.ndarray:
- """
- Run the compiled_scale_map_learner on the transformed image of dimensions
- input_image_w x input_image_h suitable to be used on such a model
-
- :param: input_image_h: The height of the input image
- :param: input_image_w: The width of the input image
- :param: transformed_image_for_depth_scale: The transformed image inputs suitable to be used as an input to the SML model
- :returns:
- sml_pred: The regression based prediction of the SML model
-
- """
-
- # Obtain the last output layer separately
- output_layer_scale_map_learner = compiled_scale_map_learner.output(0)
- x_transform, d_transform = transformed_image_for_depth_scale
-
- with torch.no_grad():
- # Perform computation like a standard OpenVINO compiled model
- sml_pred = torch.from_numpy(compiled_scale_map_learner([x_transform, d_transform])[output_layer_scale_map_learner])
- sml_pred = (
- torch.nn.functional.interpolate(
- sml_pred,
- size=(input_image_h, input_image_w),
- mode="bicubic",
- align_corners=False,
- )
- .squeeze()
- .cpu()
- .numpy()
- )
-
- return sml_pred
-
-.. code:: ipython3
-
- # Run the compiled SML model using the set of dummy inputs
- sml_pred_dummy = run_depth_scale_model(
- input_image_h=IMAGE_H,
- input_image_w=IMAGE_W,
- transformed_image_for_depth_scale=transformed_dummy_image_scale,
- compiled_scale_map_learner=compiled_scale_map_learner,
- )
-
-Storing and visualizing dummy results obtained
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-.. code:: ipython3
-
- # Base directory in which outputs would be stored as a pathlib.Path variable
- OUTPUT_DIR = Path("output")
-
- # Create the output directory adjacent to the notebook and suppress errors if the directory already exists
- OUTPUT_DIR.mkdir(exist_ok=True)
-
- # Utility functions are directly available in modules.midas.utils
- # Provide path names without any extension and let the write_depth
- # function provide them for you. Take note of the arguments.
- utils.write_depth(path=str(OUTPUT_DIR / "dummy_input"), depth=d_depth, bits=2)
- utils.write_depth(path=str(OUTPUT_DIR / "dummy_input_sml"), depth=sml_pred_dummy, bits=2)
-
-.. code:: ipython3
-
- plt.figure()
-
- img_dummy_in = mpimg.imread("data/image/dummy_img.png")
- img_dummy_out = mpimg.imread(OUTPUT_DIR / "dummy_input.png")
- img_dummy_sml_out = mpimg.imread(OUTPUT_DIR / "dummy_input_sml.png")
-
- f, axes = plt.subplots(1, 3)
- plt.subplots_adjust(right=2.0)
-
- axes[0].imshow(img_dummy_in)
- axes[1].imshow(img_dummy_out)
- axes[2].imshow(img_dummy_sml_out)
-
- axes[0].set_title("dummy input")
- axes[1].set_title("depth prediction on dummy input")
- axes[2].set_title("SML on depth estimate")
-
-
-
-
-.. parsed-literal::
-
- Text(0.5, 1.0, 'SML on depth estimate')
-
-
-
-
-.. parsed-literal::
-
-
-
-
-
-.. image:: depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_48_2.png
-
-
-Running inference on a test image
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Now role of both the dummy inputs i.e. the dummy image as well as its
-associated depth map is now over. Since we have access to the compiled
-models now, we can load the *one* image available to us for pure
-inferencing purposes and run all the above steps one by one till
-plotting of the depth map.
-
-If you haven’t noticed already the data directory of this tutorial has
-been arranged as follows. This allows us to comply to these
-`rules `__.
-
-.. code:: bash
-
- data
- ├── image
- │ ├── dummy_img.png # RGB images
- │ └── .png
- └── sparse_depth
- ├── dummy_img.png # sparse metric depth maps
- └── .png # as 16b PNG files
-
-At the same time, the depth storage method `used in the VOID
-dataset `__
-is assumed.
-
-If you are thinking of the file name format of the image for inference,
-here is the reasoning.
-
-The dataset was collected using the Intel `RealSense D435i
-camera `__, which was
-configured to produce synchronized accelerometer and gyroscope
-measurements at 400 Hz, along with synchronized VGA-size (640 x 480) RGB
-and depth streams at 30 Hz. The depth frames are acquired using active
-stereo and is aligned to the RGB frame using the sensor factory
-calibration. The frequency of sensor and depth stream input run at
-certain fixed frequencies and hence time stamping every frame captured
-is beneficial for maintaining structure as well as for debugging
-purposes later.
-
-*The image for inference and it sparse depth map is taken from the
-compressed dataset
-present*\ `here `__
-
-.. code:: ipython3
-
- # As before download the sample images for inference and take note of the image hashes if you
- # want to use them later
- download_file(
- "https://user-images.githubusercontent.com/22426058/254174393-fc6dcc5f-f677-4618-b2ef-22e8e5cb1ebe.png",
- filename="1552097950.2672.png",
- directory=Path(DATA_DIR / "image"),
- silent=True,
- )
- download_file(
- "https://user-images.githubusercontent.com/22426058/254174379-5d00b66b-57b4-4e96-91e9-36ef15ec5a0a.png",
- filename="1552097950.2672.png",
- directory=Path(DATA_DIR / "sparse_depth"),
- silent=True,
- )
-
- # Load the image and its depth scale
- img_input = data_loader.load_input_image("data/image/1552097950.2672.png")
- img_depth_input = data_loader.load_sparse_depth("data/sparse_depth/1552097950.2672.png")
-
- # Transform the input image for the depth model
- transformed_image = transform_image_for_depth(input_image=img_input, depth_model_transform=depth_model_transform)
-
- # Run the depth model on the transformed input
- depth_pred = run_depth_model(
- input_image_h=IMAGE_H,
- input_image_w=IMAGE_W,
- transformed_image=transformed_image,
- compiled_depth_model=compiled_depth_model,
- )
-
-
- # Call the function on the sparse depth map
- # with all default settings and store in appropriate variables
- int_depth, int_scales = compute_global_scale_and_shift(input_sparse_depth=img_depth_input, validity_map=None, depth_pred=depth_pred)
-
- # Transform the input image for the ScaleMapLearner model
- transformed_image_scale = transform_image_for_depth_scale(
- input_image=img_input,
- scale_map_learner_transform=scale_map_learner_transform,
- int_depth=int_depth,
- int_scales=int_scales,
- )
-
- # Run the SML model using the set of inputs
- sml_pred = run_depth_scale_model(
- input_image_h=IMAGE_H,
- input_image_w=IMAGE_W,
- transformed_image_for_depth_scale=transformed_image_scale,
- compiled_scale_map_learner=compiled_scale_map_learner,
- )
-
-
-
-.. parsed-literal::
-
- data/image/1552097950.2672.png: 0%| | 0.00/371k [00:00
-
-
-
-.. image:: depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_53_2.png
-
-
-Cleaning up the data directory
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-We will *follow suit* for the directory in which we downloaded images
-and depth maps from another repo. We shall move remove the unnecessary
-directories and files which were created during the download process.
-
-.. code:: ipython3
-
- # Remove the data directory and suppress errors(if any)
- rmtree(path=str(DATA_DIR), ignore_errors=True)
-
-Concluding notes
-~~~~~~~~~~~~~~~~
-
-
-
- 1. The code for this tutorial is adapted from the `VI-Depth
- repository `__.
- 2. Users may choose to download the original and raw datasets from
- the `VOID
- dataset `__.
- 3. The `isl-org/VI-Depth `__
- works on a slightly older version of released model assets from
- its `MiDaS sibling
- repository `__. However, the new
- releases beginning from
- `v3.1 `__
- directly have OpenVINO™ ``.xml`` and ``.bin`` model files as their
- assets thereby rendering the **major pre-processing and model
- compilation step irrelevant**.
diff --git a/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_48_2.png b/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_48_2.png
deleted file mode 100644
index 90744c0e2f9..00000000000
--- a/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_48_2.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ee26ee6b9040afd4f12332f42ac782d16f05d5126356779401404bf0eed54316
-size 215774
diff --git a/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_53_2.png b/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_53_2.png
deleted file mode 100644
index 75eae38f93a..00000000000
--- a/docs/notebooks/depth-estimation-videpth-with-output_files/depth-estimation-videpth-with-output_53_2.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b4c07380e9b4d2ed0fcf4cd3933eefa664d0cf28dbc69dcd5754474c4d4382a9
-size 190108
diff --git a/docs/notebooks/detectron2-to-openvino-with-output.rst b/docs/notebooks/detectron2-to-openvino-with-output.rst
index aa849f6178a..13983063763 100644
--- a/docs/notebooks/detectron2-to-openvino-with-output.rst
+++ b/docs/notebooks/detectron2-to-openvino-with-output.rst
@@ -60,15 +60,7 @@ Install required packages for running model
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
diff --git a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.jpg b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.jpg
index c3108b5b58f..e4d7ef7032a 100644
--- a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.jpg
+++ b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:32e589c25e3dbe3506f7daff8bf41d86686035b32d3bb85fb342937707eb7c2e
-size 58113
+oid sha256:5d10e6ad91e4f0d5c9b289d8f112f86c47143a98ce4603fcc99487dcb882d558
+size 58513
diff --git a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.png b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.png
index 46ab495c7f9..19bbb384757 100644
--- a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.png
+++ b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_22_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:1163cb4251bc3314b3a014f0a883967d75e45aef3d7cd1df2701db45c459ef61
-size 509139
+oid sha256:1a040b682a9984544752dc46565957f9cbc02d450723acb6a9fcfd9a9963ad84
+size 509276
diff --git a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.jpg b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.jpg
index 2b49c278587..3469257c99c 100644
--- a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.jpg
+++ b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:5cfdeeeb8c2dcb115730a7e3de247f28573c7d2c18d1ed88821842333e72edd7
-size 54890
+oid sha256:3fe0cf1ddc8c8ebbe68052a133d2d8dcc9478b4837f8b2e40a06696df7a04d52
+size 57234
diff --git a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.png b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.png
index 8ca760b79eb..f1bc5042ab3 100644
--- a/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.png
+++ b/docs/notebooks/detectron2-to-openvino-with-output_files/detectron2-to-openvino-with-output_32_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:c9e1a0544ca93d604126537c34e33e4be04525b1223b3cd712e812c3047838ae
-size 458489
+oid sha256:73da85bb37edfdfd377d92a77b9f588565f13dede569ce50d652e486f40c8bcd
+size 459409
diff --git a/docs/notebooks/distilbert-sequence-classification-with-output.rst b/docs/notebooks/distilbert-sequence-classification-with-output.rst
index bfdee960d39..2e777e3a8de 100644
--- a/docs/notebooks/distilbert-sequence-classification-with-output.rst
+++ b/docs/notebooks/distilbert-sequence-classification-with-output.rst
@@ -36,47 +36,31 @@ Imports
.. parsed-literal::
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
- Requirement already satisfied: openvino>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2024.0.0)
- Requirement already satisfied: transformers in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.39.3)
- Requirement already satisfied: torch>=2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.2.2+cpu)
- Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.2)
- Requirement already satisfied: numpy>=1.16.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (1.23.5)
- Requirement already satisfied: openvino-telemetry>=2023.2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (2024.1.0)
- Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (24.0)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (3.13.4)
- Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.22.2)
- Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (6.0.1)
- Requirement already satisfied: regex!=2019.12.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2024.4.16)
- Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2.31.0)
- Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.15.2)
- Requirement already satisfied: safetensors>=0.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.4.3)
- Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (4.11.0)
- Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (1.12)
- Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1)
- Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1.3)
- Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (2024.2.0)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1) (2.1.5)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.3.2)
- Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.7)
- Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2.2.1)
- Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2024.2.2)
- Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1) (1.3.0)
-
-
-.. parsed-literal::
-
+ Requirement already satisfied: openvino>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2024.1.0)
+ Requirement already satisfied: transformers in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.40.2)
+ Requirement already satisfied: torch>=2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.3.0+cpu)
+ Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (4.66.4)
+ Requirement already satisfied: numpy<2.0.0,>=1.16.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (1.23.5)
+ Requirement already satisfied: openvino-telemetry>=2023.2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (2024.1.0)
+ Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino>=2023.1.0) (24.0)
+ Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (3.14.0)
+ Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.23.0)
+ Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (6.0.1)
+ Requirement already satisfied: regex!=2019.12.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2024.4.28)
+ Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (2.31.0)
+ Requirement already satisfied: tokenizers<0.20,>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.19.1)
+ Requirement already satisfied: safetensors>=0.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from transformers) (0.4.3)
+ Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (4.11.0)
+ Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (1.12)
+ Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1)
+ Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1.4)
+ Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (2024.3.1)
+ Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1) (2.1.5)
+ Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.3.2)
+ Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (3.7)
+ Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2.2.1)
+ Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->transformers) (2024.2.2)
+ Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1) (1.3.0)
Note: you may need to restart the kernel to use updated packages.
@@ -103,6 +87,13 @@ model from Hugging Face.
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path=checkpoint)
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+
+
Initializing the Tokenizer
--------------------------
@@ -123,6 +114,13 @@ understand the context of a sentence. Here, we will use
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=checkpoint)
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+
+
Convert Model to OpenVINO Intermediate Representation format
------------------------------------------------------------
@@ -159,9 +157,9 @@ optimal execution on end-point target devices.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)
diff --git a/docs/notebooks/dolly-2-instruction-following-with-output.rst b/docs/notebooks/dolly-2-instruction-following-with-output.rst
index 170ac68c2db..51eae761dc6 100644
--- a/docs/notebooks/dolly-2-instruction-following-with-output.rst
+++ b/docs/notebooks/dolly-2-instruction-following-with-output.rst
@@ -85,14 +85,17 @@ Table of contents:
^^^^^^^^^^^^^^^^^^
- `Prerequisites <#prerequisites>`__
+- `Convert model using Optimum-CLI
+ tool <#convert-model-using-optimum-cli-tool>`__
+- `Compress model weights <#compress-model-weights>`__
- - `Select inference device <#select-inference-device>`__
-
-- `Download and Convert Model <#download-and-convert-model>`__
-
- - `NNCF model weights
- compression <#nncf-model-weights-compression>`__
+ - `Weights Compression using
+ Optimum-CLI <#weights-compression-using-optimum-cli>`__
+- `Select model variant and inference
+ device <#select-model-variant-and-inference-device>`__
+- `Instantiate Model using Optimum
+ Intel <#instantiate-model-using-optimum-intel>`__
- `Create an instruction-following inference
pipeline <#create-an-instruction-following-inference-pipeline>`__
@@ -121,16 +124,234 @@ documentation `__.
.. code:: ipython3
- %pip install -q "diffusers>=0.16.1" "transformers>=4.33.0" "torch>=2.1" "openvino>=2023.2.0" "nncf>=2.6.0" onnx "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
- %pip install -q --upgrade "git+https://github.com/huggingface/optimum-intel.git"
+ %pip unsinstall -q -y openvino openvino-dev openvino-nightly optimum optimum-intel
+ %pip install -q "diffusers>=0.16.1" "transformers>=4.33.0" "torch>=2.1" "openvino-nightly" "nncf>=2.10.0" onnx "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
+ %pip install -q "git+https://github.com/huggingface/optimum-intel.git"
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
+Convert model using Optimum-CLI tool
+------------------------------------
+
+
+
+`Optimum Intel `__ is
+the interface between the
+`Transformers `__ and
+`Diffusers `__ libraries
+and OpenVINO to accelerate end-to-end pipelines on Intel architectures.
+It provides ease-to-use cli interface for exporting models to `OpenVINO
+Intermediate Representation
+(IR) `__
+format.
+
+The command bellow demonstrates basic command for model export with
+``optimum-cli``
+
+.. code:: bash
+
+ optimum-cli export openvino --model --task
+
+where ``--model`` argument is model id from HuggingFace Hub or local
+directory with model (saved using ``.save_pretrained`` method),
+``--task`` is one of `supported
+task `__
+that exported model should solve. For LLMs it will be
+``text-generation-with-past``. If model initialization requires to use
+remote code, ``--trust-remote-code`` flag additionally should be passed.
+
+Compress model weights
+----------------------
+
+
+
+The `Weights
+Compression `__
+algorithm is aimed at compressing the weights of the models and can be
+used to optimize the model footprint and performance of large models
+where the size of weights is relatively larger than the size of
+activations, for example, Large Language Models (LLM). Compared to INT8
+compression, INT4 compression improves performance even more, but
+introduces a minor drop in prediction quality.
+
+Weights Compression using Optimum-CLI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+You can also apply fp16, 8-bit or 4-bit weight compression on the
+Linear, Convolutional and Embedding layers when exporting your model
+with the CLI by setting ``--weight-format`` to respectively fp16, int8
+or int4. This type of optimization allows to reduce the memory footprint
+and inference latency. By default the quantization scheme for int8/int4
+will be
+`asymmetric `__,
+to make it
+`symmetric `__
+you can add ``--sym``.
+
+For INT4 quantization you can also specify the following arguments : -
+The ``--group-size`` parameter will define the group size to use for
+quantization, -1 it will results in per-column quantization. - The
+``--ratio`` parameter controls the ratio between 4-bit and 8-bit
+quantization. If set to 0.9, it means that 90% of the layers will be
+quantized to int4 while 10% will be quantized to int8.
+
+Smaller group_size and ratio values usually improve accuracy at the
+sacrifice of the model size and inference latency.
+
+ **Note**: There may be no speedup for INT4/INT8 compressed models on
+ dGPU.
+
+.. code:: ipython3
+
+ from IPython.display import Markdown, display
+ import ipywidgets as widgets
+
+ prepare_int4_model = widgets.Checkbox(
+ value=True,
+ description="Prepare INT4 model",
+ disabled=False,
+ )
+ prepare_int8_model = widgets.Checkbox(
+ value=False,
+ description="Prepare INT8 model",
+ disabled=False,
+ )
+ prepare_fp16_model = widgets.Checkbox(
+ value=False,
+ description="Prepare FP16 model",
+ disabled=False,
+ )
+
+ display(prepare_int4_model)
+ display(prepare_int8_model)
+ display(prepare_fp16_model)
+
+
+
+.. parsed-literal::
+
+ Checkbox(value=True, description='Prepare INT4 model')
+
+
+
+.. parsed-literal::
+
+ Checkbox(value=False, description='Prepare INT8 model')
+
+
+
+.. parsed-literal::
+
+ Checkbox(value=False, description='Prepare FP16 model')
+
+
+.. code:: ipython3
+
+ from pathlib import Path
+
+ model_id = "databricks/dolly-v2-3b"
+ model_path = Path("dolly-v2-3b")
+
+ fp16_model_dir = model_path / "FP16"
+ int8_model_dir = model_path / "INT8_compressed_weights"
+ int4_model_dir = model_path / "INT4_compressed_weights"
+
+
+ def convert_to_fp16():
+ if (fp16_model_dir / "openvino_model.xml").exists():
+ return
+ fp16_model_dir.mkdir(parents=True, exist_ok=True)
+ export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format fp16".format(model_id)
+ export_command = export_command_base + " " + str(fp16_model_dir)
+ display(Markdown("**Export command:**"))
+ display(Markdown(f"`{export_command}`"))
+ ! $export_command
+
+
+ def convert_to_int8():
+ if (int8_model_dir / "openvino_model.xml").exists():
+ return
+ int8_model_dir.mkdir(parents=True, exist_ok=True)
+ export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int8".format(model_id)
+ export_command = export_command_base + " " + str(int8_model_dir)
+ display(Markdown("**Export command:**"))
+ display(Markdown(f"`{export_command}`"))
+ ! $export_command
+
+
+ def convert_to_int4():
+ if (int4_model_dir / "openvino_model.xml").exists():
+ return
+ int4_model_dir.mkdir(parents=True, exist_ok=True)
+ export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int4".format(model_id)
+ export_command = export_command_base + " " + str(int4_model_dir)
+ display(Markdown("**Export command:**"))
+ display(Markdown(f"`{export_command}`"))
+ ! $export_command
+
+
+ if prepare_fp16_model.value:
+ convert_to_fp16()
+ if prepare_int8_model.value:
+ convert_to_int8()
+ if prepare_int4_model.value:
+ convert_to_int4()
+
+.. code:: ipython3
+
+ fp16_weights = fp16_model_dir / "openvino_model.bin"
+ int8_weights = int8_model_dir / "openvino_model.bin"
+ int4_weights = int4_model_dir / "openvino_model.bin"
+
+ if fp16_weights.exists():
+ print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
+ for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
+ if compressed_weights.exists():
+ print(f"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB")
+ if compressed_weights.exists() and fp16_weights.exists():
+ print(f"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}")
+
+
+.. parsed-literal::
+
+ Size of model with INT4 compressed weights is 2154.54 MB
+
+
+Select model variant and inference device
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
select device from dropdown list for running inference using OpenVINO
+.. code:: ipython3
+
+ available_models = []
+ if int4_model_dir.exists():
+ available_models.append("INT4")
+ if int8_model_dir.exists():
+ available_models.append("INT8")
+ if fp16_model_dir.exists():
+ available_models.append("FP16")
+
+ model_to_run = widgets.Dropdown(
+ options=available_models,
+ value=available_models[0],
+ description="Model to run:",
+ disabled=False,
+ )
+
+ model_to_run
+
+
+
+
+.. parsed-literal::
+
+ Dropdown(description='Model to run:', options=('INT4',), value='INT4')
+
+
+
.. code:: ipython3
import ipywidgets as widgets
@@ -152,12 +373,12 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
- Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')
+ Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')
-Download and Convert Model
---------------------------
+Instantiate Model using Optimum Intel
+-------------------------------------
@@ -179,14 +400,18 @@ Below is an example of the Dolly model
model_id = "databricks/dolly-v2-3b"
-model = AutoModelForCausalLM.from_pretrained(model_id)
- +model = OVModelForCausalLM.from_pretrained(model_id, from_transformers=True)
+ +model = OVModelForCausalLM.from_pretrained(model_id, export=True)
Model class initialization starts with calling ``from_pretrained``
method. When downloading and converting Transformers model, the
-parameter ``export=True`` should be added. For models where size more We
-can save the converted model for the next usage with the
-``save_pretrained`` method. Tokenizer class and pipelines API are
-compatible with Optimum models.
+parameter ``export=True`` should be added (as we already converted model
+before, we do not need to provide this parameter). We can save the
+converted model for the next usage with the ``save_pretrained`` method.
+Tokenizer class and pipelines API are compatible with Optimum models.
+
+You can find more details about OpenVINO LLM inference using HuggingFace
+Optimum API in `LLM inference
+guide `__.
.. code:: ipython3
@@ -194,27 +419,21 @@ compatible with Optimum models.
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
- model_id = "databricks/dolly-v2-3b"
- model_path = Path("dolly-v2-3b")
+ if model_to_run.value == "INT4":
+ model_dir = int4_model_dir
+ elif model_to_run.value == "INT8":
+ model_dir = int8_model_dir
+ else:
+ model_dir = fp16_model_dir
+ print(f"Loading model from {model_dir}")
- tokenizer = AutoTokenizer.from_pretrained(model_id)
+ tokenizer = AutoTokenizer.from_pretrained(model_dir)
current_device = device.value
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
- if model_path.exists():
- ov_model = OVModelForCausalLM.from_pretrained(model_path, device=current_device, ov_config=ov_config)
- else:
- ov_model = OVModelForCausalLM.from_pretrained(
- model_id,
- device=current_device,
- export=True,
- ov_config=ov_config,
- load_in_8bit=False,
- )
- ov_model.half()
- ov_model.save_pretrained(model_path)
+ ov_model = OVModelForCausalLM.from_pretrained(model_dir, device=current_device, ov_config=ov_config)
.. parsed-literal::
@@ -225,87 +444,35 @@ compatible with Optimum models.
.. parsed-literal::
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
- 2023-11-17 13:10:43.359093: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2023-11-17 13:10:43.398436: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-01 10:43:29.010748: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-01 10:43:29.012724: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-05-01 10:43:29.047558: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-05-01 10:43:29.048434: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
- 2023-11-17 13:10:44.026743: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
- Compiling the model to CPU ...
-
-
-NNCF model weights compression
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-NNCF `Weights Compression
-algorithm `__
-compresses weights of a model to ``INT8``. This is an alternative to
-`Quantization
-algorithm `__
-that compresses both weights and activations. Weight compression is
-effective in optimizing footprint and performance of large models where
-the size of weights is significantly larger than the size of
-activations, for example, in Large Language Models (LLMs) such as Dolly
-2.0. Additionally, Weight Compression usually leads to almost no
-accuracy drop.
-
-.. code:: ipython3
-
- to_compress = widgets.Checkbox(
- value=True,
- description="INT8 Compression",
- disabled=False,
- )
- print("Click on checkbox for enabling / disabling weights compression")
- to_compress
+ 2024-05-01 10:43:29.742257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
+ warn("The installed version of bitsandbytes was compiled without GPU support. "
.. parsed-literal::
- Click on checkbox for enabling / disabling weights compression
-
-
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
.. parsed-literal::
- Checkbox(value=True, description='INT8 Compression')
-
-
-
-.. code:: ipython3
-
- import gc
- from optimum.intel import OVQuantizer, OVConfig, OVWeightQuantizationConfig
-
- compressed_model_path = Path(f"{model_path}_compressed")
-
-
- def calculate_compression_rate(model_path_ov, model_path_ov_compressed):
- model_size_original = model_path_ov.with_suffix(".bin").stat().st_size / 2**20
- model_size_compressed = model_path_ov_compressed.with_suffix(".bin").stat().st_size / 2**20
- print(f"* Original IR model size: {model_size_original:.2f} MB")
- print(f"* Compressed IR model size: {model_size_compressed:.2f} MB")
- print(f"* Model compression rate: {model_size_original / model_size_compressed:.3f}")
-
-
- if to_compress.value:
- if not compressed_model_path.exists():
- quantizer = OVQuantizer.from_pretrained(ov_model)
- ov_config = OVConfig(quantization_config=OVWeightQuantizationConfig(bits=8))
- quantizer.quantize(save_directory=compressed_model_path, ov_config=ov_config)
- del quantizer
- gc.collect()
-
- calculate_compression_rate(model_path / "openvino_model.xml", compressed_model_path / "openvino_model.xml")
- ov_model = OVModelForCausalLM.from_pretrained(compressed_model_path, device=current_device, ov_config=ov_config)
+ WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
+ PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
+ Python 3.8.18 (you have 3.8.10)
+ Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
+ Memory-efficient attention, SwiGLU, sparse and more won't be available.
+ Set XFORMERS_MORE_DETAILS=1 for more details
+ Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
.. parsed-literal::
- * Original IR model size: 5297.21 MB
- * Compressed IR model size: 2657.89 MB
- * Model compression rate: 1.993
+ Loading model from dolly-v2-3b/INT4_compressed_weights
.. parsed-literal::
diff --git a/docs/notebooks/efficient-sam-with-output.rst b/docs/notebooks/efficient-sam-with-output.rst
index 4f7c2cde7e1..ffaaa51f8b1 100644
--- a/docs/notebooks/efficient-sam-with-output.rst
+++ b/docs/notebooks/efficient-sam-with-output.rst
@@ -84,15 +84,7 @@ Prerequisites
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -110,398 +102,13 @@ Prerequisites
.. parsed-literal::
Cloning into 'EfficientSAM'...
-
-
-.. parsed-literal::
-
remote: Enumerating objects: 424, done.[K
- remote: Counting objects: 1% (1/85)[K
-remote: Counting objects: 2% (2/85)[K
-remote: Counting objects: 3% (3/85)[K
-remote: Counting objects: 4% (4/85)[K
-remote: Counting objects: 5% (5/85)[K
-remote: Counting objects: 7% (6/85)[K
-remote: Counting objects: 8% (7/85)[K
-remote: Counting objects: 9% (8/85)[K
-remote: Counting objects: 10% (9/85)[K
-remote: Counting objects: 11% (10/85)[K
-remote: Counting objects: 12% (11/85)[K
-remote: Counting objects: 14% (12/85)[K
-remote: Counting objects: 15% (13/85)[K
-remote: Counting objects: 16% (14/85)[K
-remote: Counting objects: 17% (15/85)[K
-remote: Counting objects: 18% (16/85)[K
-remote: Counting objects: 20% (17/85)[K
-remote: Counting objects: 21% (18/85)[K
-remote: Counting objects: 22% (19/85)[K
-remote: Counting objects: 23% (20/85)[K
-remote: Counting objects: 24% (21/85)[K
-remote: Counting objects: 25% (22/85)[K
-remote: Counting objects: 27% (23/85)[K
-remote: Counting objects: 28% (24/85)[K
-remote: Counting objects: 29% (25/85)[K
-remote: Counting objects: 30% (26/85)[K
-remote: Counting objects: 31% (27/85)[K
-remote: Counting objects: 32% (28/85)[K
-remote: Counting objects: 34% (29/85)[K
-remote: Counting objects: 35% (30/85)[K
-remote: Counting objects: 36% (31/85)[K
-remote: Counting objects: 37% (32/85)[K
-remote: Counting objects: 38% (33/85)[K
-remote: Counting objects: 40% (34/85)[K
-remote: Counting objects: 41% (35/85)[K
-remote: Counting objects: 42% (36/85)[K
-remote: Counting objects: 43% (37/85)[K
-remote: Counting objects: 44% (38/85)[K
-remote: Counting objects: 45% (39/85)[K
-remote: Counting objects: 47% (40/85)[K
-remote: Counting objects: 48% (41/85)[K
-remote: Counting objects: 49% (42/85)[K
-remote: Counting objects: 50% (43/85)[K
-remote: Counting objects: 51% (44/85)[K
-remote: Counting objects: 52% (45/85)[K
-remote: Counting objects: 54% (46/85)[K
-remote: Counting objects: 55% (47/85)[K
-remote: Counting objects: 56% (48/85)[K
-remote: Counting objects: 57% (49/85)[K
-remote: Counting objects: 58% (50/85)[K
-remote: Counting objects: 60% (51/85)[K
-remote: Counting objects: 61% (52/85)[K
-remote: Counting objects: 62% (53/85)[K
-remote: Counting objects: 63% (54/85)[K
-remote: Counting objects: 64% (55/85)[K
-remote: Counting objects: 65% (56/85)[K
-remote: Counting objects: 67% (57/85)[K
-remote: Counting objects: 68% (58/85)[K
-remote: Counting objects: 69% (59/85)[K
-remote: Counting objects: 70% (60/85)[K
-remote: Counting objects: 71% (61/85)[K
-remote: Counting objects: 72% (62/85)[K
-remote: Counting objects: 74% (63/85)[K
-remote: Counting objects: 75% (64/85)[K
-remote: Counting objects: 76% (65/85)[K
-remote: Counting objects: 77% (66/85)[K
-remote: Counting objects: 78% (67/85)[K
-remote: Counting objects: 80% (68/85)[K
-remote: Counting objects: 81% (69/85)[K
-remote: Counting objects: 82% (70/85)[K
-remote: Counting objects: 83% (71/85)[K
-remote: Counting objects: 84% (72/85)[K
-remote: Counting objects: 85% (73/85)[K
-remote: Counting objects: 87% (74/85)[K
-remote: Counting objects: 88% (75/85)[K
-remote: Counting objects: 89% (76/85)[K
-remote: Counting objects: 90% (77/85)[K
-remote: Counting objects: 91% (78/85)[K
-remote: Counting objects: 92% (79/85)[K
-remote: Counting objects: 94% (80/85)[K
-remote: Counting objects: 95% (81/85)[K
-remote: Counting objects: 96% (82/85)[K
-remote: Counting objects: 97% (83/85)[K
-remote: Counting objects: 98% (84/85)[K
-remote: Counting objects: 100% (85/85)[K
-remote: Counting objects: 100% (85/85), done.[K
- remote: Compressing objects: 3% (1/33)[K
-remote: Compressing objects: 6% (2/33)[K
-remote: Compressing objects: 9% (3/33)[K
-remote: Compressing objects: 12% (4/33)[K
-remote: Compressing objects: 15% (5/33)[K
-remote: Compressing objects: 18% (6/33)[K
-remote: Compressing objects: 21% (7/33)[K
-remote: Compressing objects: 24% (8/33)[K
-remote: Compressing objects: 27% (9/33)[K
-remote: Compressing objects: 30% (10/33)[K
-remote: Compressing objects: 33% (11/33)[K
-remote: Compressing objects: 36% (12/33)[K
-remote: Compressing objects: 39% (13/33)[K
-remote: Compressing objects: 42% (14/33)[K
-remote: Compressing objects: 45% (15/33)[K
-remote: Compressing objects: 48% (16/33)[K
-remote: Compressing objects: 51% (17/33)[K
-remote: Compressing objects: 54% (18/33)[K
-remote: Compressing objects: 57% (19/33)[K
-remote: Compressing objects: 60% (20/33)[K
-remote: Compressing objects: 63% (21/33)[K
-remote: Compressing objects: 66% (22/33)[K
-remote: Compressing objects: 69% (23/33)[K
-remote: Compressing objects: 72% (24/33)[K
-remote: Compressing objects: 75% (25/33)[K
-remote: Compressing objects: 78% (26/33)[K
-remote: Compressing objects: 81% (27/33)[K
-remote: Compressing objects: 84% (28/33)[K
-remote: Compressing objects: 87% (29/33)[K
-remote: Compressing objects: 90% (30/33)[K
-remote: Compressing objects: 93% (31/33)[K
-remote: Compressing objects: 96% (32/33)[K
-remote: Compressing objects: 100% (33/33)[K
-remote: Compressing objects: 100% (33/33), done.[K
- Receiving objects: 0% (1/424)
-Receiving objects: 1% (5/424)
-
-.. parsed-literal::
-
- Receiving objects: 2% (9/424)
-Receiving objects: 3% (13/424)
-Receiving objects: 4% (17/424)
-Receiving objects: 5% (22/424)
-
-.. parsed-literal::
-
- Receiving objects: 5% (24/424), 7.11 MiB | 7.01 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 5% (24/424), 15.89 MiB | 7.84 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 5% (24/424), 25.21 MiB | 8.33 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 5% (24/424), 35.32 MiB | 8.76 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 6% (26/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 7% (30/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 8% (34/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 9% (39/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 10% (43/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 11% (47/424), 35.32 MiB | 8.76 MiB/s
-Receiving objects: 12% (51/424), 35.32 MiB | 8.76 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 12% (54/424), 46.28 MiB | 9.54 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 12% (54/424), 58.26 MiB | 10.40 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 12% (54/424), 70.74 MiB | 11.14 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 13% (56/424), 70.74 MiB | 11.14 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 13% (56/424), 81.30 MiB | 11.36 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 13% (56/424), 93.08 MiB | 11.64 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 13% (56/424), 105.71 MiB | 11.89 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 14% (60/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 15% (64/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 16% (68/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 17% (73/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 18% (77/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 19% (81/424), 105.71 MiB | 11.89 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 20% (85/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 21% (90/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 22% (94/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 23% (98/424), 105.71 MiB | 11.89 MiB/s
-Receiving objects: 24% (102/424), 105.71 MiB | 11.89 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 25% (106/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 26% (111/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 27% (115/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 28% (119/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 29% (123/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 30% (128/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 31% (132/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 32% (136/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 33% (140/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 34% (145/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 35% (149/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 36% (153/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 37% (157/424), 112.21 MiB | 11.99 MiB/s
-Receiving objects: 38% (162/424), 112.21 MiB | 11.99 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 118.79 MiB | 12.04 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 132.56 MiB | 12.57 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 147.18 MiB | 13.34 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 159.14 MiB | 13.27 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 172.36 MiB | 13.30 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 186.46 MiB | 13.47 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 38% (164/424), 201.36 MiB | 13.62 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 39% (166/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 40% (170/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 41% (174/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 42% (179/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 43% (183/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 44% (187/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 45% (191/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 46% (196/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 47% (200/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 48% (204/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 49% (208/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 50% (212/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 51% (217/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 52% (221/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 53% (225/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 54% (229/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 55% (234/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 56% (238/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 57% (242/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 58% (246/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 59% (251/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 60% (255/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 61% (259/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 62% (263/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 63% (268/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 64% (272/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 65% (276/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 66% (280/424), 208.90 MiB | 13.65 MiB/s
-Receiving objects: 67% (285/424), 208.90 MiB | 13.65 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 67% (288/424), 216.43 MiB | 13.99 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 67% (288/424), 232.14 MiB | 14.78 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 68% (289/424), 240.34 MiB | 15.09 MiB/s
-Receiving objects: 69% (293/424), 240.34 MiB | 15.09 MiB/s
-Receiving objects: 70% (297/424), 240.34 MiB | 15.09 MiB/s
-Receiving objects: 71% (302/424), 240.34 MiB | 15.09 MiB/s
-Receiving objects: 72% (306/424), 240.34 MiB | 15.09 MiB/s
-Receiving objects: 73% (310/424), 240.34 MiB | 15.09 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 73% (310/424), 248.86 MiB | 15.43 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 74% (314/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 75% (318/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 76% (323/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 77% (327/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 78% (331/424), 257.49 MiB | 15.77 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 79% (335/424), 257.49 MiB | 15.77 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 80% (340/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 81% (344/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 82% (348/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 83% (352/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 84% (357/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 85% (361/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 86% (365/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 87% (369/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 88% (374/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 89% (378/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 90% (382/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 91% (386/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 92% (391/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 93% (395/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 94% (399/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 95% (403/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 96% (408/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 97% (412/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 98% (416/424), 257.49 MiB | 15.77 MiB/s
-remote: Total 424 (delta 76), reused 52 (delta 52), pack-reused 339[K
- Receiving objects: 99% (420/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 100% (424/424), 257.49 MiB | 15.77 MiB/s
-Receiving objects: 100% (424/424), 262.14 MiB | 12.58 MiB/s, done.
- Resolving deltas: 0% (0/246)
-Resolving deltas: 4% (10/246)
-Resolving deltas: 6% (17/246)
-Resolving deltas: 14% (36/246)
-Resolving deltas: 18% (46/246)
-Resolving deltas: 22% (56/246)
-Resolving deltas: 23% (57/246)
-Resolving deltas: 26% (64/246)
-Resolving deltas: 27% (68/246)
-Resolving deltas: 32% (81/246)
-Resolving deltas: 36% (89/246)
-Resolving deltas: 37% (92/246)
-Resolving deltas: 41% (101/246)
-
-.. parsed-literal::
-
- Resolving deltas: 42% (105/246)
-Resolving deltas: 44% (110/246)
-Resolving deltas: 46% (114/246)
-
-.. parsed-literal::
-
- Resolving deltas: 48% (120/246)
-Resolving deltas: 49% (122/246)
-Resolving deltas: 52% (128/246)
-Resolving deltas: 54% (134/246)
-Resolving deltas: 58% (143/246)
-Resolving deltas: 62% (153/246)
-Resolving deltas: 63% (155/246)
-Resolving deltas: 66% (163/246)
-Resolving deltas: 67% (165/246)
-Resolving deltas: 69% (172/246)
-Resolving deltas: 70% (174/246)
-Resolving deltas: 88% (217/246)
-Resolving deltas: 96% (237/246)
-Resolving deltas: 97% (240/246)
-Resolving deltas: 98% (243/246)
-
-.. parsed-literal::
-
- Resolving deltas: 99% (245/246)
-
-.. parsed-literal::
-
- Resolving deltas: 100% (246/246)
-Resolving deltas: 100% (246/246), done.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM
+ remote: Counting objects: 100% (85/85), done.[K
+ remote: Compressing objects: 100% (33/33), done.[K
+ remote: Total 424 (delta 76), reused 52 (delta 52), pack-reused 339[K
+ Receiving objects: 100% (424/424), 262.14 MiB | 24.49 MiB/s, done.
+ Resolving deltas: 100% (246/246), done.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM
Load PyTorch model
@@ -756,27 +363,23 @@ disk using ``openvino.save_model``.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:220: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:220: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if (
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:163: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:163: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
size = int(math.sqrt(xy_num))
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert size * size == xy_num
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:166: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:166: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if size != h or size != w:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:251: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam_encoder.py:251: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert x.shape[2] == num_patches
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if num_pts > self.decoder_max_num_input_points:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:92: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:92: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
elif num_pts < self.decoder_max_num_input_points:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:126: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/efficient-sam/EfficientSAM/efficient_sam/efficient_sam.py:126: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if output_w > 0 and output_h > 0:
@@ -1040,14 +643,10 @@ architecture type, we should specify ``transformer`` in ``model_type``.
.. parsed-literal::
- 2024-04-17 23:43:02.085337: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:43:02.119379: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:01:29.399785: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:01:29.433596: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:43:02.732513: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-07 00:01:30.076398: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@@ -1095,10 +694,6 @@ architecture type, we should specify ``transformer`` in ``model_type``.
.. parsed-literal::
INFO:nncf:57 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:88 ignored nodes were found by name in the NNCFGraph
@@ -1123,12 +718,6 @@ architecture type, we should specify ``transformer`` in ``model_type``.
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
- return Tensor(self.data * unwrap_tensor_data(other))
-
-
.. parsed-literal::
@@ -1224,8 +813,8 @@ Compare quantized model size
.. parsed-literal::
Size of FP16 model is 21.50 MB
- Size of INT8 quantized model is 10.96 MB
- Compression rate for INT8 model: 1.962
+ Size of INT8 quantized model is 11.08 MB
+ Compression rate for INT8 model: 1.941
Compare inference time of the FP16 and INT8 models
@@ -1251,22 +840,18 @@ models, we use ``bencmark_app``.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 42.86 ms
+ [ INFO ] Read model took 40.36 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] batched_images (node: batched_images) : f32 / [...] / [?,?,?,?]
@@ -1286,11 +871,7 @@ models, we use ``bencmark_app``.
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_2) : f32 / [...] / [?,?,?]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 1414.09 ms
+ [ INFO ] Compile model took 1355.34 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@@ -1298,10 +879,6 @@ models, we use ``bencmark_app``.
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ] MULTI_DEVICE_PRIORITIES: CPU
-
-
-.. parsed-literal::
-
[ INFO ] CPU:
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
@@ -1315,6 +892,7 @@ models, we use ``bencmark_app``.
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@@ -1324,6 +902,7 @@ models, we use ``bencmark_app``.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'batched_images'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'batched_points'!. This input will be filled with random values!
@@ -1333,25 +912,17 @@ models, we use ``bencmark_app``.
[ INFO ] Fill input 'batched_point_labels' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in full mode (inputs filling are included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 644.24 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 645.58 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 49 iterations
- [ INFO ] Duration: 15719.86 ms
+ [ INFO ] Duration: 15813.12 ms
[ INFO ] Latency:
- [ INFO ] Median: 1890.94 ms
- [ INFO ] Average: 1870.83 ms
- [ INFO ] Min: 622.00 ms
- [ INFO ] Max: 1963.97 ms
- [ INFO ] Throughput: 3.12 FPS
+ [ INFO ] Median: 1908.46 ms
+ [ INFO ] Average: 1882.77 ms
+ [ INFO ] Min: 609.20 ms
+ [ INFO ] Max: 1971.02 ms
+ [ INFO ] Throughput: 3.10 FPS
.. code:: ipython3
@@ -1366,30 +937,22 @@ models, we use ``bencmark_app``.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 66.17 ms
+ [ INFO ] Read model took 53.68 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] batched_images (node: batched_images) : f32 / [...] / [?,?,?,?]
[ INFO ] batched_points (node: batched_points) : i64 / [...] / [?,?,?,?]
-
-
-.. parsed-literal::
-
[ INFO ] batched_point_labels (node: batched_point_labels) : i64 / [...] / [?,?,?]
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
@@ -1405,11 +968,7 @@ models, we use ``bencmark_app``.
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_3) : f32 / [...] / [?,?,?,?,?]
[ INFO ] ***NO_NAME*** (node: aten::reshape/Reshape_2) : f32 / [...] / [?,?,?]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 1891.51 ms
+ [ INFO ] Compile model took 1854.44 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@@ -1430,6 +989,7 @@ models, we use ``bencmark_app``.
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@@ -1439,6 +999,7 @@ models, we use ``bencmark_app``.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'batched_images'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'batched_points'!. This input will be filled with random values!
@@ -1448,25 +1009,17 @@ models, we use ``bencmark_app``.
[ INFO ] Fill input 'batched_point_labels' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in full mode (inputs filling are included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 588.40 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 596.95 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 55 iterations
- [ INFO ] Duration: 16253.02 ms
+ [ INFO ] Duration: 16443.47 ms
[ INFO ] Latency:
- [ INFO ] Median: 1752.95 ms
- [ INFO ] Average: 1732.80 ms
- [ INFO ] Min: 518.84 ms
- [ INFO ] Max: 1804.26 ms
- [ INFO ] Throughput: 3.38 FPS
+ [ INFO ] Median: 1775.29 ms
+ [ INFO ] Average: 1756.44 ms
+ [ INFO ] Min: 630.26 ms
+ [ INFO ] Max: 1849.40 ms
+ [ INFO ] Throughput: 3.34 FPS
Interactive segmentation demo
diff --git a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_16_1.png b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_16_1.png
index cedf338d0b6..f3bcf9b59bd 100644
--- a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_16_1.png
+++ b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_16_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:53d92c7dd4acfc54d070d038a1b8a86ea51c1033ba17518ea266a7081901086f
-size 1260793
+oid sha256:f283354c61f2d5c9aea2fce146bd130ae9133059c0d90b6fadea3cb6207466da
+size 1261054
diff --git a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_24_1.png b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_24_1.png
index 9a50592e4e2..82bb5364f01 100644
--- a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_24_1.png
+++ b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_24_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:dda127dcb26fadf81397d1872fc64e0a6af73cccd791912bfac525cb9040c24f
-size 1260102
+oid sha256:c36f8c8a366405e85e109c43647bb0cf0f0a92d10d7ddeb70c19e87a6818485b
+size 1259311
diff --git a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_35_1.png b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_35_1.png
index fa0d5c0961d..0b05fa9506f 100644
--- a/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_35_1.png
+++ b/docs/notebooks/efficient-sam-with-output_files/efficient-sam-with-output_35_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:e5ca7e36b0b567092b0960c0ce5d48df585bdf87175f045909d4665325dc7de5
-size 1263513
+oid sha256:0979318f5d2a5bdb8e7deac91c7ab319d24efcca8be6e5612c4dca27a802ca95
+size 1262900
diff --git a/docs/notebooks/encodec-audio-compression-with-output.rst b/docs/notebooks/encodec-audio-compression-with-output.rst
index 0446a02865b..34e9cac2630 100644
--- a/docs/notebooks/encodec-audio-compression-with-output.rst
+++ b/docs/notebooks/encodec-audio-compression-with-output.rst
@@ -59,11 +59,6 @@ Install required dependencies:
%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu "openvino>=2023.3.0" "torch>=2.1" "torchaudio>=2.1" "encodec>=0.1.1" "gradio>=4.19" "librosa>=0.8.1" "matplotlib<=3.7" tqdm
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
@@ -134,7 +129,7 @@ bandwidth.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
@@ -294,7 +289,7 @@ similar as possible to the original.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
@@ -394,13 +389,13 @@ with ``ov.save_model``.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:60: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:60: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
ideal_length = (math.ceil(n_frames) - 1) * stride + (kernel_size - padding_total)
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
max_pad = max(padding_left, padding_right)
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if length <= max_pad:
@@ -420,11 +415,11 @@ with ``ov.save_model``.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:358: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:358: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
quantized_out = torch.tensor(0.0, device=q_indices.device)
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:359: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/quantization/core_vq.py:359: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
for i, indices in enumerate(q_indices):
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:103: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/encodec/modules/conv.py:103: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (padding_left + padding_right) <= x.shape[-1]
diff --git a/docs/notebooks/fast-segment-anything-with-output.rst b/docs/notebooks/fast-segment-anything-with-output.rst
index 1880f1a574a..c67be7550a3 100644
--- a/docs/notebooks/fast-segment-anything-with-output.rst
+++ b/docs/notebooks/fast-segment-anything-with-output.rst
@@ -75,25 +75,8 @@ Install requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -106,29 +89,29 @@ Imports
import ipywidgets as widgets
from pathlib import Path
-
+
import openvino as ov
import torch
from PIL import Image, ImageDraw
from ultralytics import FastSAM
-
+
# Fetch skip_kernel_extension module
import requests
-
+
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
)
open("skip_kernel_extension.py", "w").write(r.text)
# Fetch `notebook_utils` module
import requests
-
+
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
-
+
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
-
+
%load_ext skip_kernel_extension
FastSAM in Ultralytics
@@ -148,7 +131,7 @@ model and generate a segmentation map.
model_name = "FastSAM-x"
model = FastSAM(model_name)
-
+
# Run inference on an image
image_uri = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg"
image_uri = download_file(image_uri)
@@ -162,92 +145,7 @@ model and generate a segmentation map.
.. parsed-literal::
-
- 0%| | 0.00/138M [00:00 None:
ov_config = ov_config or {}
self.model = core.compile_model(ov_model, device, ov_config)
-
+
self.stride = stride
self.pt = False
self.fp16 = False
self.names = {0: "object"}
-
+
def __call__(self, im, **_):
result = self.model(im)
return torch.from_numpy(result[0]), torch.from_numpy(result[1])
@@ -424,7 +298,7 @@ pipeline.
ov_config = {}
if "GPU" in device.value or ("AUTO" in device.value and "GPU" in core.available_devices):
ov_config = {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"}
-
+
wrapped_model = OVWrapper(
ov_model_path,
device=device.value,
@@ -432,23 +306,15 @@ pipeline.
ov_config=ov_config,
)
model.predictor.model = wrapped_model
-
+
ov_results = model(image_uri, device=device.value, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9)
-
-
-
-
-
.. parsed-literal::
- image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 1024x1024 42 objects, 504.2ms
-
-
-.. parsed-literal::
-
- Speed: 6.5ms preprocess, 504.2ms inference, 31.9ms postprocess per image at shape (1, 3, 1024, 1024)
+
+ image 1/1 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/fast-segment-anything/coco_bike.jpg: 1024x1024 42 objects, 510.3ms
+ Speed: 6.8ms preprocess, 510.3ms inference, 34.2ms postprocess per image at shape (1, 3, 1024, 1024)
One can observe the converted model outputs in the next cell, they is
@@ -489,7 +355,7 @@ The optimization process contains the following steps:
description="Quantization",
disabled=False,
)
-
+
do_quantize
@@ -525,20 +391,20 @@ repo <../yolov8-optimization/>`__.
.. code:: ipython3
%%skip not $do_quantize.value
-
+
import pickle
from contextlib import contextmanager
from zipfile import ZipFile
-
+
import cv2
from tqdm.autonotebook import tqdm
-
+
import nncf
-
-
+
+
COLLECT_CALIBRATION_DATA = False
calibration_data = []
-
+
@contextmanager
def calibration_data_collection():
global COLLECT_CALIBRATION_DATA
@@ -547,58 +413,58 @@ repo <../yolov8-optimization/>`__.
yield
finally:
COLLECT_CALIBRATION_DATA = False
-
-
+
+
class NNCFWrapper:
def __init__(self, ov_model, stride=32) -> None:
self.model = core.read_model(ov_model)
self.compiled_model = core.compile_model(self.model, device_name="CPU")
-
+
self.stride = stride
self.pt = False
self.fp16 = False
self.names = {0: "object"}
-
+
def __call__(self, im, **_):
if COLLECT_CALIBRATION_DATA:
calibration_data.append(im)
-
+
result = self.compiled_model(im)
return torch.from_numpy(result[0]), torch.from_numpy(result[1])
-
+
# Fetch data from the web and descibe a dataloader
DATA_URL = "https://ultralytics.com/assets/coco128.zip"
OUT_DIR = Path('.')
-
+
download_file(DATA_URL, directory=OUT_DIR, show_progress=True)
-
+
if not (OUT_DIR / "coco128/images/train2017").exists():
with ZipFile('coco128.zip', "r") as zip_ref:
zip_ref.extractall(OUT_DIR)
-
+
class COCOLoader(torch.utils.data.Dataset):
def __init__(self, images_path):
self.images = list(Path(images_path).iterdir())
-
+
def __getitem__(self, index):
if isinstance(index, slice):
return [self.read_image(image_path) for image_path in self.images[index]]
return self.read_image(self.images[index])
-
+
def read_image(self, image_path):
image = cv2.imread(str(image_path))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
return image
-
+
def __len__(self):
return len(self.images)
-
-
+
+
def collect_calibration_data_for_decoder(model, calibration_dataset_size: int,
calibration_cache_path: Path):
global calibration_data
-
-
+
+
if not calibration_cache_path.exists():
coco_dataset = COCOLoader(OUT_DIR / 'coco128/images/train2017')
with calibration_data_collection():
@@ -610,10 +476,10 @@ repo <../yolov8-optimization/>`__.
else:
with open(calibration_cache_path, "rb") as f:
calibration_data = pickle.load(f)
-
+
return calibration_data
-
-
+
+
def quantize(model, save_model_path: Path, calibration_cache_path: Path,
calibration_dataset_size: int, preset: nncf.QuantizationPreset):
calibration_data = collect_calibration_data_for_decoder(
@@ -634,10 +500,10 @@ repo <../yolov8-optimization/>`__.
)
)
ov.save_model(quantized_ov_decoder, save_model_path)
-
+
wrapped_model = NNCFWrapper(ov_model_path, stride=model.predictor.model.stride)
model.predictor.model = wrapped_model
-
+
calibration_dataset_size = 128
quantized_model_path = Path(f"{model_name}_quantized") / "FastSAM-x.xml"
calibration_cache_path = Path(f"calibration_data/coco{calibration_dataset_size}.pkl")
@@ -667,48 +533,16 @@ repo <../yolov8-optimization/>`__.
.. parsed-literal::
INFO:nncf:3 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:8 ignored nodes were found by types in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:Not adding activation input quantizer for operation: 275 __module.model.22/aten::sigmoid/Sigmoid
-
-
-.. parsed-literal::
-
INFO:nncf:Not adding activation input quantizer for operation: 325 __module.model.22.dfl.conv/aten::_convolution/Convolution
-
-
-.. parsed-literal::
-
INFO:nncf:Not adding activation input quantizer for operation: 351 __module.model.22/aten::sub/Subtract
-
-
-.. parsed-literal::
-
INFO:nncf:Not adding activation input quantizer for operation: 352 __module.model.22/aten::add/Add
-
-
-.. parsed-literal::
-
INFO:nncf:Not adding activation input quantizer for operation: 365 __module.model.22/aten::add/Add_1
378 __module.model.22/aten::div/Divide
-
-
-
-.. parsed-literal::
-
+
INFO:nncf:Not adding activation input quantizer for operation: 366 __module.model.22/aten::sub/Subtract_1
-
-
-.. parsed-literal::
-
- INFO:nncf:Not adding activation input quantizer for operation: 388 __module.model.22/aten::mul/Multiply
+ INFO:nncf:Not adding activation input quantizer for operation: 389 __module.model.22/aten::mul/Multiply
@@ -732,12 +566,6 @@ repo <../yolov8-optimization/>`__.
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/nncf/experimental/tensor/tensor.py:84: RuntimeWarning: invalid value encountered in multiply
- return Tensor(self.data * unwrap_tensor_data(other))
-
-
.. parsed-literal::
@@ -770,15 +598,15 @@ calibration dataset to measure the performance.
.. code:: ipython3
%%skip not $do_quantize.value
-
+
import datetime
-
+
coco_dataset = COCOLoader(OUT_DIR / 'coco128/images/train2017')
calibration_dataset_size = 128
-
+
wrapped_model = OVWrapper(ov_model_path, device=device.value, stride=model.predictor.model.stride)
model.predictor.model = wrapped_model
-
+
start_time = datetime.datetime.now()
for image in tqdm(coco_dataset, desc="Measuring inference time"):
model(image, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9, verbose=False)
@@ -802,10 +630,10 @@ calibration dataset to measure the performance.
.. code:: ipython3
%%skip not $do_quantize.value
-
+
quantized_wrapped_model = OVWrapper(quantized_model_path, device=device.value, stride=model.predictor.model.stride)
model.predictor.model = quantized_wrapped_model
-
+
start_time = datetime.datetime.now()
for image in tqdm(coco_dataset, desc="Measuring inference time"):
model(image, retina_masks=True, imgsz=1024, conf=0.6, iou=0.9, verbose=False)
@@ -823,9 +651,9 @@ calibration dataset to measure the performance.
.. parsed-literal::
- Segmented in 23 seconds
- Resulting in 5.57 fps
- That is 2.96 times faster!
+ Segmented in 22 seconds
+ Resulting in 5.82 fps
+ That is 3.09 times faster!
Try out the converted pipeline
@@ -845,8 +673,8 @@ bounding boxes on input image.
import cv2
import numpy as np
import matplotlib.pyplot as plt
-
-
+
+
def fast_process(
annotations,
image,
@@ -859,12 +687,12 @@ bounding boxes on input image.
):
original_h = image.height
original_w = image.width
-
+
if better_quality:
for i, mask in enumerate(annotations):
mask = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_CLOSE, np.ones((3, 3), np.uint8))
annotations[i] = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_OPEN, np.ones((8, 8), np.uint8))
-
+
inner_mask = fast_show_mask(
annotations,
plt.gca(),
@@ -874,7 +702,7 @@ bounding boxes on input image.
target_height=original_h,
target_width=original_w,
)
-
+
if with_contours:
contour_all = []
temp = np.zeros((original_h, original_w, 1))
@@ -892,18 +720,18 @@ bounding boxes on input image.
cv2.drawContours(temp, contour_all, -1, (255, 255, 255), 2 // scale)
color = np.array([0 / 255, 0 / 255, 255 / 255, 0.9])
contour_mask = temp / 255 * color.reshape(1, 1, -1)
-
+
image = image.convert("RGBA")
overlay_inner = Image.fromarray((inner_mask * 255).astype(np.uint8), "RGBA")
image.paste(overlay_inner, (0, 0), overlay_inner)
-
+
if with_contours:
overlay_contour = Image.fromarray((contour_mask * 255).astype(np.uint8), "RGBA")
image.paste(overlay_contour, (0, 0), overlay_contour)
-
+
return image
-
-
+
+
# CPU post process
def fast_show_mask(
annotation,
@@ -921,7 +749,7 @@ bounding boxes on input image.
areas = np.sum(annotation, axis=(1, 2))
sorted_indices = np.argsort(areas)[::1]
annotation = annotation[sorted_indices]
-
+
index = (annotation != 0).argmax(axis=0)
if random_color:
color = np.random.random((mask_sum, 1, 1, 3))
@@ -930,32 +758,32 @@ bounding boxes on input image.
transparency = np.ones((mask_sum, 1, 1, 1)) * 0.6
visual = np.concatenate([color, transparency], axis=-1)
mask_image = np.expand_dims(annotation, -1) * visual
-
+
mask = np.zeros((height, weight, 4))
-
+
h_indices, w_indices = np.meshgrid(np.arange(height), np.arange(weight), indexing="ij")
indices = (index[h_indices, w_indices], h_indices, w_indices, slice(None))
-
+
mask[h_indices, w_indices, :] = mask_image[indices]
if bbox is not None:
x1, y1, x2, y2 = bbox
ax.add_patch(plt.Rectangle((x1, y1), x2 - x1, y2 - y1, fill=False, edgecolor="b", linewidth=1))
-
+
if not retinamask:
mask = cv2.resize(mask, (target_width, target_height), interpolation=cv2.INTER_NEAREST)
-
+
return mask
.. code:: ipython3
import gradio as gr
-
+
examples = [
[image_uri],
["https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/empty_road_mapillary.jpg"],
["https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/wall.jpg"],
]
-
+
object_points = []
background_points = []
bbox_points = []
@@ -981,14 +809,14 @@ based on user input.
model.predictor.model = quantized_wrapped_model
else:
model.predictor.model = wrapped_model
-
+
input_size = int(input_size)
w, h = image.size
scale = input_size / max(w, h)
new_w = int(w * scale)
new_h = int(h * scale)
image = image.resize((new_w, new_h))
-
+
results = model(
image,
retina_masks=use_retina,
@@ -996,14 +824,14 @@ based on user input.
conf=conf_threshold,
imgsz=input_size,
)
-
+
masks = results[0].masks.data
# Calculate annotations
if not (object_points or bbox_points):
annotations = masks.cpu().numpy()
else:
annotations = []
-
+
if object_points:
all_points = object_points + background_points
labels = [1] * len(object_points) + [0] * len(background_points)
@@ -1026,20 +854,20 @@ based on user input.
x = max(min(x, new_w), 0)
y = max(min(y, new_h), 0)
scaled_bbox_points.append((x, y))
-
+
for i in range(0, len(scaled_bbox_points) - 1, 2):
x0, y0, x1, y1 = *scaled_bbox_points[i], *scaled_bbox_points[i + 1]
-
+
intersection_area = torch.sum(masks[:, y0:y1, x0:x1], dim=(1, 2))
masks_area = torch.sum(masks, dim=(1, 2))
bbox_area = (y1 - y0) * (x1 - x0)
-
+
union = bbox_area + masks_area - intersection_area
iou = intersection_area / union
max_iou_index = torch.argmax(iou)
-
+
annotations.append(masks[max_iou_index].cpu().numpy())
-
+
return fast_process(
annotations=np.array(annotations),
image=image,
@@ -1085,8 +913,8 @@ based on user input.
fill=color,
)
return img
-
-
+
+
def clear_points() -> (Image.Image, None):
"""Gradio clear points callback."""
global object_points, background_points, bbox_points
@@ -1095,8 +923,8 @@ based on user input.
background_points = []
bbox_points = []
return last_image, None
-
-
+
+
def save_last_picked_image(img: Image.Image) -> None:
"""Gradio callback saves the last used image."""
global last_image
@@ -1106,8 +934,8 @@ based on user input.
clear_points()
# Removes the segmentation map output
return None
-
-
+
+
with gr.Blocks(title="Fast SAM") as demo:
with gr.Row(variant="panel"):
original_img = gr.Image(label="Input", value=examples[0][0], type="pil")
@@ -1133,18 +961,18 @@ based on user input.
run_on_click=True,
outputs=segmented_img,
)
-
+
# Callbacks
original_img.select(select_point, inputs=[original_img, point_type], outputs=original_img)
original_img.upload(save_last_picked_image, inputs=original_img, outputs=segmented_img)
clear_button.click(clear_points, outputs=[original_img, segmented_img])
segment_button.click(segment, inputs=[original_img, model_type], outputs=segmented_img)
-
+
try:
demo.queue().launch(debug=False)
except Exception:
demo.queue().launch(share=True, debug=False)
-
+
# If you are launching remotely, specify server_name and server_port
# EXAMPLE: `demo.launch(server_name="your server name", server_port="server port in int")`
# To learn more please refer to the Gradio docs: https://gradio.app/docs/
@@ -1153,7 +981,7 @@ based on user input.
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
-
+
To create a public link, set `share=True` in `launch()`.
diff --git a/docs/notebooks/fastcomposer-image-generation-with-output.rst b/docs/notebooks/fastcomposer-image-generation-with-output.rst
deleted file mode 100644
index 67294a27a84..00000000000
--- a/docs/notebooks/fastcomposer-image-generation-with-output.rst
+++ /dev/null
@@ -1,1177 +0,0 @@
-`FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention `__
-=====================================================================================================================
-
-FastComposer uses subject embeddings extracted by an image encoder to
-augment the generic text conditioning in diffusion models, enabling
-personalized image generation based on subject images and textual
-instructions with only forward passes. Moreover it addresses two
-problems:
-
-- **The identity blending problem.** To address the problem in the
- multi-subject generation it proposes cross-attention localization
- supervision during training, enforcing the attention of reference
- subjects localized to the correct regions in the target images.
-
-- **Subject overfitting.** Naively conditioning on subject embeddings
- results in subject overfitting. FastComposer proposes delayed subject
- conditioning in the denoising step to maintain both identity and
- editability in subject-driven image generation.
-
-FastComposer generates images of multiple unseen individuals with
-different styles, actions, and contexts.
-
- **NOTE**: ``model.py`` is slightly changed ``model.py`` from
- fastcomposer repository. There are two main changes: - some unused
- lines of code are removed to avoid errors if there are no CUDA
- drivers in the system - changes to have compatibility with
- transformers >= 4.30.1 (due to security vulnerability)
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Install Prerequisites <#install-prerequisites>`__
-- `Convert models to OpenVINO Intermediate representation (IR)
- format <#convert-models-to-openvino-intermediate-representation-ir-format>`__
-
- - `Convert text_encoder <#convert-text_encoder>`__
- - `The Object Transform <#the-object-transform>`__
- - `The Image Encoder <#the-image-encoder>`__
- - `Postfuse module <#postfuse-module>`__
- - `Convert Unet <#convert-unet>`__
-
-- `Rebuild pipeline <#rebuild-pipeline>`__
-- `Inference <#inference>`__
-- `Run Gradio <#run-gradio>`__
-
-.. container:: alert alert-block alert-warning
-
- ::
-
- This tutorial requires about 25-28GB of free memory to generate one image. Each extra image requires ~11GB of free memory.
-
-Install Prerequisites
----------------------
-
- Install required packages.
-
-.. code:: ipython3
-
- %pip install -q --upgrade pip
- %pip install -q --extra-index-url https://download.pytorch.org/whl/cpu torch torchvision
- %pip install -q transformers huggingface-hub accelerate "diffusers>=0.16.1" "gradio>=4.19"
- %pip install -q "openvino>=2023.1.0"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
- Note: you may need to restart the kernel to use updated packages.
- Note: you may need to restart the kernel to use updated packages.
- Note: you may need to restart the kernel to use updated packages.
-
-
-Clone FastComposer project from GitHub
-
-.. code:: ipython3
-
- from pathlib import Path
-
-
- # clone FastComposer repo
- if not Path("fastcomposer").exists():
- !git clone https://github.com/mit-han-lab/fastcomposer.git
- else:
- print("FastComposer repo already cloned")
-
-
-.. parsed-literal::
-
- FastComposer repo already cloned
-
-
-Download pretrained model.
-
-.. code:: ipython3
-
- from huggingface_hub import hf_hub_download
-
-
- model_path = hf_hub_download(repo_id="mit-han-lab/fastcomposer", filename="pytorch_model.bin")
-
-Convert models to OpenVINO Intermediate representation (IR) format
-------------------------------------------------------------------
-
-
-
-Define a configuration and make instance of ``FastComposerModel``.
-
-.. code:: ipython3
-
- from model import FastComposerModel
- from dataclasses import dataclass
-
- import torch
-
-
- @dataclass()
- class Config:
- finetuned_model_path = str(model_path)
- image_encoder_name_or_path = "openai/clip-vit-large-patch14"
- localization_layers = 5
- mask_loss = False
- mask_loss_prob = 0.5
- non_ema_revision = None
- object_localization = False
- object_localization_weight = 0.01
- object_resolution = 256
- pretrained_model_name_or_path = "runwayml/stable-diffusion-v1-5"
- revision = None
-
-
- config = Config()
- model = FastComposerModel.from_pretrained(config)
- model.load_state_dict(torch.load(config.finetuned_model_path, map_location="cpu"), strict=False)
-
-
-.. parsed-literal::
-
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
- torch.utils._pytree._register_pytree_node(
- WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
- PyTorch 2.1.0+cu121 with CUDA 1201 (you have 2.2.0+cu121)
- Python 3.8.18 (you have 3.8.10)
- Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
- Memory-efficient attention, SwiGLU, sparse and more won't be available.
- Set XFORMERS_MORE_DETAILS=1 for more details
- 2024-02-22 11:01:58.013035: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-02-22 11:01:58.014759: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
- 2024-02-22 11:01:58.051348: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
- To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
- 2024-02-22 11:01:58.839838: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
- torch.utils._pytree._register_pytree_node(
-
-
-
-
-.. parsed-literal::
-
- _IncompatibleKeys(missing_keys=['vae.encoder.mid_block.attentions.0.to_q.weight', 'vae.encoder.mid_block.attentions.0.to_q.bias', 'vae.encoder.mid_block.attentions.0.to_k.weight', 'vae.encoder.mid_block.attentions.0.to_k.bias', 'vae.encoder.mid_block.attentions.0.to_v.weight', 'vae.encoder.mid_block.attentions.0.to_v.bias', 'vae.encoder.mid_block.attentions.0.to_out.0.weight', 'vae.encoder.mid_block.attentions.0.to_out.0.bias', 'vae.decoder.mid_block.attentions.0.to_q.weight', 'vae.decoder.mid_block.attentions.0.to_q.bias', 'vae.decoder.mid_block.attentions.0.to_k.weight', 'vae.decoder.mid_block.attentions.0.to_k.bias', 'vae.decoder.mid_block.attentions.0.to_v.weight', 'vae.decoder.mid_block.attentions.0.to_v.bias', 'vae.decoder.mid_block.attentions.0.to_out.0.weight', 'vae.decoder.mid_block.attentions.0.to_out.0.bias'], unexpected_keys=['text_encoder.embeddings.position_ids', 'image_encoder.vision_model.embeddings.position_ids', 'vae.encoder.mid_block.attentions.0.query.weight', 'vae.encoder.mid_block.attentions.0.query.bias', 'vae.encoder.mid_block.attentions.0.key.weight', 'vae.encoder.mid_block.attentions.0.key.bias', 'vae.encoder.mid_block.attentions.0.value.weight', 'vae.encoder.mid_block.attentions.0.value.bias', 'vae.encoder.mid_block.attentions.0.proj_attn.weight', 'vae.encoder.mid_block.attentions.0.proj_attn.bias', 'vae.decoder.mid_block.attentions.0.query.weight', 'vae.decoder.mid_block.attentions.0.query.bias', 'vae.decoder.mid_block.attentions.0.key.weight', 'vae.decoder.mid_block.attentions.0.key.bias', 'vae.decoder.mid_block.attentions.0.value.weight', 'vae.decoder.mid_block.attentions.0.value.bias', 'vae.decoder.mid_block.attentions.0.proj_attn.weight', 'vae.decoder.mid_block.attentions.0.proj_attn.bias'])
-
-
-
-Pipeline consist of next models: ``Unet``, ``TextEncoder``,
-``ImageEncoder`` and ``PostfuseModule`` (MLP), ``object_transforms`` .
-
-.. figure:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/1d858a65-e7c7-43f8-83df-1e896d745725
- :alt: inference-pipeline
-
- inference-pipeline
-
-So, convert the models into OpenVINO IR format.
-
-Convert text_encoder
-~~~~~~~~~~~~~~~~~~~~
-
-
-
-Model components are PyTorch modules, that can be converted with
-``ov.convert_model`` function directly. We also use ``ov.save_model``
-function to serialize the result of conversion. Let’s create a helper
-function.
-
-.. code:: ipython3
-
- import gc
- import openvino as ov
-
-
- def convert(model: torch.nn.Module, xml_path: str, example_input):
- xml_path = Path(xml_path)
- if not xml_path.exists():
- xml_path.parent.mkdir(parents=True, exist_ok=True)
- with torch.no_grad():
- converted_model = ov.convert_model(model, example_input=example_input)
- ov.save_model(converted_model, xml_path)
-
- # cleanup memory
- torch._C._jit_clear_class_registry()
- torch.jit._recursive.concrete_type_store = torch.jit._recursive.ConcreteTypeStore()
- torch.jit._state._clear_class_state()
-
-The text encoder is responsible for converting the input prompt into an
-embedding space that can be fed to the next stage’s U-Net. Typically, it
-is a transformer-based encoder that maps a sequence of input tokens to a
-sequence of text embeddings.
-
-The input for the text encoder consists of a tensor ``input_ids``, which
-contains token indices from the text processed by the tokenizer and
-padded to the maximum length accepted by the model.
-
-.. code:: ipython3
-
- text_encoder_ir_xml_path = Path("models/text_encoder_ir.xml")
- example_input = torch.zeros((1, 77), dtype=torch.int64)
-
- model.text_encoder.eval()
- convert(model.text_encoder, text_encoder_ir_xml_path, example_input)
-
- del model.text_encoder
- gc.collect();
-
-
-.. parsed-literal::
-
- WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
-
-
-.. parsed-literal::
-
- [ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:273: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:281: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:313: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
-
-
-The Object Transform
-~~~~~~~~~~~~~~~~~~~~
-
-
-
-It pads an incoming user image to square and resize it. An input is a
-tensor of size [3, height, width].
-
-.. code:: ipython3
-
- from collections import OrderedDict
- from torchvision import transforms as T
- from fastcomposer.fastcomposer.transforms import PadToSquare
-
-
- object_transforms = torch.nn.Sequential(
- OrderedDict(
- [
- ("pad_to_square", PadToSquare(fill=0, padding_mode="constant")),
- (
- "resize",
- T.Resize(
- (config.object_resolution, config.object_resolution),
- interpolation=T.InterpolationMode.BILINEAR,
- antialias=True,
- ),
- ),
- ("convert_to_float", T.ConvertImageDtype(torch.float32)),
- ]
- )
- )
-
- object_transforms_ir_xml_path = Path("models/object_transforms_ir.xml")
- example_input = torch.zeros([3, 1500, 1453], dtype=torch.uint8)
-
- object_transforms.eval()
- convert(object_transforms, object_transforms_ir_xml_path, example_input)
-
- del object_transforms
- gc.collect();
-
-
-.. parsed-literal::
-
- /home/ea/work/openvino_notebooks/notebooks/fastcomposer-image-generation/fastcomposer/fastcomposer/transforms.py:35: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if h == w:
- /home/ea/work/openvino_notebooks/notebooks/fastcomposer-image-generation/fastcomposer/fastcomposer/transforms.py:37: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- elif h > w:
-
-
-The Image Encoder
-~~~~~~~~~~~~~~~~~
-
-
-
-The image encoder is a CLIP (Contrastive Language-Image Pretraining)
-Image Encoder. It takes a transformed image from the previous step as
-input and transforms it into a high-dimensional vector or embeddings.
-
-.. code:: ipython3
-
- image_encoder_ir_xml_path = Path("models/image_encoder_ir.xml")
- example_input = torch.zeros((1, 2, 3, 256, 256), dtype=torch.float32)
-
- model.image_encoder.eval()
- convert(model.image_encoder, image_encoder_ir_xml_path, example_input)
-
- del model.image_encoder
- gc.collect();
-
-
-.. parsed-literal::
-
- /home/ea/work/openvino_notebooks/notebooks/fastcomposer-image-generation/model.py:108: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if h != self.image_size or w != self.image_size:
-
-
-Postfuse module
-~~~~~~~~~~~~~~~
-
-
-
-On this step it is employed a multilayer perceptron (MLP) to augment the
-text embeddings with visual features extracted from the reference
-subjects. The Postfuse module concatenates the word embeddings with the
-visual features and feeds the resulting augmented embeddings into the
-MLP.
-
-.. code:: ipython3
-
- postfuse_module_ir_xml_path = Path("models/postfuse_module_ir.xml")
-
- example_input = [
- torch.zeros((1, 77, 768), dtype=torch.float32),
- torch.zeros((1, 2, 1, 768), dtype=torch.float32),
- torch.zeros((1, 77), dtype=torch.bool),
- torch.zeros((1,), dtype=torch.int64),
- ]
-
- model.postfuse_module.eval()
- convert(model.postfuse_module, postfuse_module_ir_xml_path, example_input)
-
- del model.postfuse_module
- gc.collect();
-
-Convert Unet
-~~~~~~~~~~~~
-
-
-
-U-Net model gradually denoises latent image representation guided by
-text encoder hidden state.
-
-.. code:: ipython3
-
- unet_ir_xml_path = Path("models/unet_ir.xml")
-
- example_input = [
- torch.zeros((8, 4, 64, 64), dtype=torch.float32),
- torch.zeros((), dtype=torch.int64),
- torch.zeros((8, 77, 768), dtype=torch.float32),
- ]
- model.unet.eval()
- convert(model.unet, unet_ir_xml_path, example_input)
-
-
- del model
- del example_input
-
- gc.collect()
-
-
-.. parsed-literal::
-
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/models/unet_2d_condition.py:915: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if dim % default_overall_up_factor != 0:
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/peft/tuners/loha/layer.py:303: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
- def forward(ctx, w1a, w1b, w2a, w2b, scale=torch.tensor(1)):
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/peft/tuners/loha/layer.py:326: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
- def forward(ctx, t1, w1a, w1b, t2, w2a, w2b, scale=torch.tensor(1)):
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/models/downsampling.py:135: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- assert hidden_states.shape[1] == self.channels
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/models/downsampling.py:144: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- assert hidden_states.shape[1] == self.channels
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- assert hidden_states.shape[1] == self.channels
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if hidden_states.shape[0] >= 64:
-
-
-
-
-.. parsed-literal::
-
- 16724
-
-
-
-Rebuild pipeline
-----------------
-
-
-
-Also, it needs to modify some internal FastComposer entities, to use
-OpenVINO models. First of all, how to get results. For example, to
-convert outputs from numpy to torch types.
-
-.. code:: ipython3
-
- import numpy as np
- from diffusers.pipelines.stable_diffusion import StableDiffusionPipelineOutput
- from diffusers.pipelines.stable_diffusion import StableDiffusionPipeline
- from diffusers.loaders import TextualInversionLoaderMixin
- from typing import Any, Callable, Dict, List, Optional, Union
- from PIL import Image
-
-
- class StableDiffusionFastCompposerPipeline(StableDiffusionPipeline):
- r"""
- Pipeline for text-to-image generation using FastComposer (https://arxiv.org/abs/2305.10431).
-
- This model inherits from [`StableDiffusionPipeline`]. Check the superclass documentation for the generic methods the
- library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)
- """
-
- @torch.no_grad()
- def _tokenize_and_mask_noun_phrases_ends(self, caption):
- input_ids = self.special_tokenizer.encode(caption)
- noun_phrase_end_mask = [False for _ in input_ids]
- clean_input_ids = []
- clean_index = 0
-
- for i, id in enumerate(input_ids):
- if id == self.image_token_id:
- noun_phrase_end_mask[clean_index - 1] = True
- else:
- clean_input_ids.append(id)
- clean_index += 1
-
- max_len = self.special_tokenizer.model_max_length
-
- if len(clean_input_ids) > max_len:
- clean_input_ids = clean_input_ids[:max_len]
- else:
- clean_input_ids = clean_input_ids + [self.tokenizer.pad_token_id] * (max_len - len(clean_input_ids))
-
- if len(noun_phrase_end_mask) > max_len:
- noun_phrase_end_mask = noun_phrase_end_mask[:max_len]
- else:
- noun_phrase_end_mask = noun_phrase_end_mask + [False] * (max_len - len(noun_phrase_end_mask))
-
- clean_input_ids = torch.tensor(clean_input_ids, dtype=torch.long)
- noun_phrase_end_mask = torch.tensor(noun_phrase_end_mask, dtype=torch.bool)
- return clean_input_ids.unsqueeze(0), noun_phrase_end_mask.unsqueeze(0)
-
- @torch.no_grad()
- def _encode_augmented_prompt(
- self,
- prompt: str,
- reference_images: List[Image.Image],
- device: torch.device,
- weight_dtype: torch.dtype,
- ):
- # TODO: check this
- # encode reference images
- object_pixel_values = []
- for image in reference_images:
- image_tensor = torch.from_numpy(np.array(image.convert("RGB"))).permute(2, 0, 1)
- image = torch.from_numpy((self.object_transforms(image_tensor)[0]))
- object_pixel_values.append(image)
-
- object_pixel_values = torch.stack(object_pixel_values, dim=0).to(memory_format=torch.contiguous_format).float()
- object_pixel_values = object_pixel_values.unsqueeze(0).to(dtype=torch.float32, device=device)
- object_embeds = self.image_encoder(object_pixel_values)[0]
- object_embeds = torch.from_numpy(object_embeds)
-
- # augment the text embedding
- input_ids, image_token_mask = self._tokenize_and_mask_noun_phrases_ends(prompt)
- input_ids, image_token_mask = input_ids.to(device), image_token_mask.to(device)
-
- num_objects = image_token_mask.sum(dim=1)
-
- text_embeds = torch.from_numpy(self.text_encoder(input_ids)[0])
- augmented_prompt_embeds = self.postfuse_module([text_embeds, object_embeds, image_token_mask, num_objects])[0]
- return torch.from_numpy(augmented_prompt_embeds)
-
- def _encode_prompt(
- self,
- prompt,
- device,
- num_images_per_prompt,
- do_classifier_free_guidance,
- negative_prompt=None,
- ):
- r"""
- Encodes the prompt into text encoder hidden states.
-
- Args:
- prompt (`str` or `List[str]`, *optional*):
- prompt to be encoded
- device: (`torch.device`):
- torch device
- num_images_per_prompt (`int`):
- number of images that should be generated per prompt
- do_classifier_free_guidance (`bool`):
- whether to use classifier free guidance or not
- negative_prompt (`str` or `List[str]`, *optional*):
- The prompt or prompts not to guide the image generation. If not defined, one has to pass
- `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is
- less than `1`).
- """
- if isinstance(prompt, str):
- batch_size = 1
- elif isinstance(prompt, list):
- batch_size = len(prompt)
-
- # textual inversion: procecss multi-vector tokens if necessary
- if isinstance(self, TextualInversionLoaderMixin):
- prompt = self.maybe_convert_prompt(prompt, self.tokenizer)
-
- text_inputs = self.tokenizer(
- prompt,
- padding="max_length",
- max_length=self.tokenizer.model_max_length,
- truncation=True,
- return_tensors="pt",
- )
- text_input_ids = text_inputs.input_ids
- untruncated_ids = self.tokenizer(prompt, padding="longest", return_tensors="pt").input_ids
-
- if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(text_input_ids, untruncated_ids):
- removed_text = self.tokenizer.batch_decode(untruncated_ids[:, self.tokenizer.model_max_length - 1 : -1])
- print(
- "The following part of your input was truncated because CLIP can only handle sequences up to"
- f" {self.tokenizer.model_max_length} tokens: {removed_text}"
- )
-
- prompt_embeds = self.text_encoder(text_input_ids.to(device))[0]
- prompt_embeds = torch.from_numpy(prompt_embeds)
-
- bs_embed, seq_len, _ = prompt_embeds.shape
- # duplicate text embeddings for each generation per prompt, using mps friendly method
- prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
- prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)
-
- # get unconditional embeddings for classifier free guidance
- if do_classifier_free_guidance:
- uncond_tokens: List[str]
- if negative_prompt is None:
- uncond_tokens = [""] * batch_size
- elif type(prompt) is not type(negative_prompt):
- raise TypeError(f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !=" f" {type(prompt)}.")
- elif isinstance(negative_prompt, str):
- uncond_tokens = [negative_prompt]
- elif batch_size != len(negative_prompt):
- raise ValueError(
- f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:"
- f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches"
- " the batch size of `prompt`."
- )
- else:
- uncond_tokens = negative_prompt
-
- # textual inversion: procecss multi-vector tokens if necessary
- if isinstance(self, TextualInversionLoaderMixin):
- uncond_tokens = self.maybe_convert_prompt(uncond_tokens, self.tokenizer)
-
- max_length = prompt_embeds.shape[1]
- uncond_input = self.tokenizer(
- uncond_tokens,
- padding="max_length",
- max_length=max_length,
- truncation=True,
- return_tensors="pt",
- )
-
- negative_prompt_embeds = self.text_encoder(uncond_input.input_ids.to(device))[0]
- negative_prompt_embeds = torch.from_numpy(negative_prompt_embeds)
-
- if do_classifier_free_guidance:
- # duplicate unconditional embeddings for each generation per prompt, using mps friendly method
- seq_len = negative_prompt_embeds.shape[1]
-
- negative_prompt_embeds = negative_prompt_embeds.to(dtype=torch.float32, device=device)
-
- negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)
- negative_prompt_embeds = negative_prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)
-
- # For classifier free guidance, we need to do two forward passes.
- # Here we concatenate the unconditional and text embeddings into a single batch
- # to avoid doing two forward passes
- prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds])
-
- return prompt_embeds
-
- @torch.no_grad()
- def __call__(
- self,
- prompt: Union[str, List[str]] = None,
- height: Optional[int] = None,
- width: Optional[int] = None,
- num_inference_steps: int = 50,
- guidance_scale: float = 7.5,
- negative_prompt: Optional[Union[str, List[str]]] = None,
- num_images_per_prompt: Optional[int] = 1,
- eta: float = 0.0,
- generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
- latents: Optional[torch.FloatTensor] = None,
- prompt_embeds: Optional[torch.FloatTensor] = None,
- negative_prompt_embeds: Optional[torch.FloatTensor] = None,
- output_type: Optional[str] = "pil",
- return_dict: bool = True,
- callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,
- callback_steps: int = 1,
- cross_attention_kwargs: Optional[Dict[str, Any]] = None,
- alpha_: float = 0.7,
- reference_subject_images: List[Image.Image] = None,
- augmented_prompt_embeds: Optional[torch.FloatTensor] = None,
- ):
- r"""
- Function invoked when calling the pipeline for generation.
-
- Args:
- prompt (`str` or `List[str]`, *optional*):
- The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.
- instead.
- height (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor):
- The height in pixels of the generated image.
- width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor):
- The width in pixels of the generated image.
- num_inference_steps (`int`, *optional*, defaults to 50):
- The number of denoising steps. More denoising steps usually lead to a higher quality image at the
- expense of slower inference.
- guidance_scale (`float`, *optional*, defaults to 7.5):
- Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
- `guidance_scale` is defined as `w` of equation 2. of [Imagen
- Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
- 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
- usually at the expense of lower image quality.
- negative_prompt (`str` or `List[str]`, *optional*):
- The prompt or prompts not to guide the image generation. If not defined, one has to pass
- `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is
- less than `1`).
- num_images_per_prompt (`int`, *optional*, defaults to 1):_unwrap_model
- The number of images to generate per prompt.
- eta (`float`, *optional*, defaults to 0.0):
- Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
- [`schedulers.DDIMScheduler`], will be ignored for others.
- generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
- One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
- to make generation deterministic.
- latents (`torch.FloatTensor`, *optional*):
- Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
- generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
- tensor will ge generated by sampling using the supplied random `generator`.
- prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
- provided, text embeddings will be generated from `prompt` input argument.
- negative_prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
- weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input
- argument.
- output_type (`str`, *optional*, defaults to `"pil"`):
- The output format of the generate image. Choose between
- [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
- return_dict (`bool`, *optional*, defaults to `True`):
- Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a
- plain tuple.
- callback (`Callable`, *optional*):
- A function that will be called every `callback_steps` steps during inference. The function will be
- called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
- callback_steps (`int`, *optional*, defaults to 1):
- The frequency at which the `callback` function will be called. If not specified, the callback will be
- called at every step.
- cross_attention_kwargs (`dict`, *optional*):
- A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under
- `self.processor` in
- [diffusers.cross_attention](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/cross_attention.py).
- alpha_ (`float`, defaults to 0.7):
- The ratio of subject conditioning. If `alpha_` is 0.7, the beginning 30% of denoising steps use text prompts, while the
- last 70% utilize image-augmented prompts. Increase alpha for identity preservation, decrease it for prompt consistency.
- reference_subject_images (`List[PIL.Image.Image]`):
- a list of PIL images that are used as reference subjects. The number of images should be equal to the number of augmented
- tokens in the prompts.
- augmented_prompt_embeds: (`torch.FloatTensor`, *optional*):
- Pre-generated image augmented text embeddings. If not provided, embeddings will be generated from `prompt` and
- `reference_subject_images`.
- Examples:
-
- Returns:
- [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`:
- [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] if `return_dict` is True, otherwise a `tuple.
- When returning a tuple, the first element is a list with the generated images, and the second element is a
- list of `bool`s denoting whether the corresponding generated image likely represents "not-safe-for-work"
- (nsfw) content, according to the `safety_checker`.
- """
- # 0. Default height and width to unet
- height = height or self.unet.config.sample_size * self.vae_scale_factor
- width = width or self.unet.config.sample_size * self.vae_scale_factor
-
- # 1. Check inputs. Raise error if not correct
- self.check_inputs(
- prompt,
- height,
- width,
- callback_steps,
- negative_prompt,
- prompt_embeds,
- negative_prompt_embeds,
- )
-
- assert (prompt is not None and reference_subject_images is not None) or (
- prompt_embeds is not None and augmented_prompt_embeds is not None
- ), "Prompt and reference subject images or prompt_embeds and augmented_prompt_embeds must be provided."
-
- # 2. Define call parameters
- if prompt is not None and isinstance(prompt, str):
- batch_size = 1
- elif prompt is not None and isinstance(prompt, list):
- batch_size = len(prompt)
- else:
- batch_size = prompt_embeds.shape[0]
-
- device = self._execution_device
- # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
- # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
- # corresponds to doing no classifier free guidance.
- do_classifier_free_guidance = guidance_scale > 1.0
-
- assert do_classifier_free_guidance
-
- # 3. Encode input prompt
- prompt_text_only = prompt.replace("", "")
-
- prompt_embeds = self._encode_prompt(
- prompt_text_only,
- device,
- num_images_per_prompt,
- do_classifier_free_guidance,
- negative_prompt,
- )
-
- if augmented_prompt_embeds is None:
- augmented_prompt_embeds = self._encode_augmented_prompt(prompt, reference_subject_images, device, prompt_embeds.dtype)
- augmented_prompt_embeds = augmented_prompt_embeds.repeat(num_images_per_prompt, 1, 1)
-
- prompt_embeds = torch.cat([prompt_embeds, augmented_prompt_embeds], dim=0)
-
- # 4. Prepare timesteps
- self.scheduler.set_timesteps(num_inference_steps, device=device)
- timesteps = self.scheduler.timesteps
-
- # 5. Prepare latent variables
- # num_channels_latents = self.unet.in_channels
- num_channels_latents = 4
- latents = self.prepare_latents(
- batch_size * num_images_per_prompt,
- num_channels_latents,
- height,
- width,
- prompt_embeds.dtype,
- device,
- generator,
- latents,
- )
-
- start_subject_conditioning_step = (1 - alpha_) * num_inference_steps
-
- extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
- (
- null_prompt_embeds,
- text_prompt_embeds,
- augmented_prompt_embeds,
- ) = prompt_embeds.chunk(3)
-
- # 7. Denoising loop
- num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order
- with self.progress_bar(total=num_inference_steps) as progress_bar:
- for i, t in enumerate(timesteps):
- latent_model_input = torch.cat([latents] * 2) if do_classifier_free_guidance else latents
- latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)
-
- if i <= start_subject_conditioning_step:
- current_prompt_embeds = torch.cat([null_prompt_embeds, text_prompt_embeds], dim=0)
- else:
- current_prompt_embeds = torch.cat([null_prompt_embeds, augmented_prompt_embeds], dim=0)
-
- # predict the noise residual
- noise_pred = self.unet(
- [
- latent_model_input,
- t,
- current_prompt_embeds,
- # cross_attention_kwargs
- ],
- )[0]
- noise_pred = torch.from_numpy(noise_pred)
-
- # perform guidance
- if do_classifier_free_guidance:
- noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)
- noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)
- else:
- assert 0, "Not Implemented"
-
- # compute the previous noisy sample x_t -> x_t-1
- latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs).prev_sample
-
- # call the callback, if provided
- if i == len(timesteps) - 1 or ((i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0):
- progress_bar.update()
- if callback is not None and i % callback_steps == 0:
- callback(i, t, latents)
-
- if output_type == "latent":
- image = latents
- has_nsfw_concept = None
- elif output_type == "pil":
- # 8. Post-processing
- image = self.decode_latents(latents)
-
- # 9. Run safety checker
- image, has_nsfw_concept = self.run_safety_checker(image, device, prompt_embeds.dtype)
-
- # 10. Convert to PIL
- image = self.numpy_to_pil(image)
- else:
- # 8. Post-processing
- image = self.decode_latents(latents)
-
- # 9. Run safety checker
- image, has_nsfw_concept = self.run_safety_checker(image, device, prompt_embeds.dtype)
-
- # Offload last model to CPU
- if hasattr(self, "final_offload_hook") and self.final_offload_hook is not None:
- self.final_offload_hook.offload()
-
- if not return_dict:
- return (image, has_nsfw_concept)
-
- return StableDiffusionPipelineOutput(images=image, nsfw_content_detected=has_nsfw_concept)
-
-And replace all model in the pipeline by converted models.
-
-.. code:: ipython3
-
- import PIL
- from transformers import CLIPTokenizer
-
-
- def create_pipeline(
- args,
- *,
- text_encoder,
- image_encoder,
- unet,
- object_transforms,
- postfuse_module,
- ):
- weight_dtype = torch.float32
-
- tokenizer = CLIPTokenizer.from_pretrained(
- args.pretrained_model_name_or_path,
- subfolder="tokenizer",
- revision=args.revision,
- )
- tokenizer.add_tokens(["img"], special_tokens=True)
- image_token_id = tokenizer.convert_tokens_to_ids("img")
-
- pipe = StableDiffusionFastCompposerPipeline.from_pretrained(args.pretrained_model_name_or_path, torch_dtype=weight_dtype)
-
- pipe.object_transforms = object_transforms
- pipe.unet = unet
- pipe.text_encoder = text_encoder
- pipe.postfuse_module = postfuse_module
- pipe.image_encoder = image_encoder
- pipe.image_token_id = image_token_id
- pipe.special_tokenizer = tokenizer
-
- return pipe
-
-
- class ModelWrapper:
- def __init__(self, model):
- super().__init__()
- self.model = model
-
- def inference(
- self,
- image1: PIL.Image.Image,
- image2: PIL.Image.Image,
- prompt: str,
- negative_prompt: str,
- seed: int,
- guidance_scale: float,
- alpha_: float,
- num_steps: int,
- num_images: int,
- ):
- print("Running model inference...")
- image = []
- if image1 is not None:
- image.append(image1)
-
- if image2 is not None:
- image.append(image2)
-
- if len(image) == 0:
- return [], "You need to upload at least one image."
-
- num_subject_in_text = (np.array(self.model.special_tokenizer.encode(prompt)) == self.model.image_token_id).sum()
- if num_subject_in_text != len(image):
- return (
- [],
- f"Number of subjects in the text description doesn't match the number of reference images, #text subjects: {num_subject_in_text} #reference image: {len(image)}",
- )
-
- if seed == -1:
- seed = np.random.randint(0, 1000000)
-
- generator = torch.manual_seed(seed)
-
- return (
- self.model(
- prompt=prompt,
- negative_prompt=negative_prompt,
- height=512,
- width=512,
- num_inference_steps=num_steps,
- guidance_scale=guidance_scale,
- num_images_per_prompt=num_images,
- generator=generator,
- alpha_=alpha_,
- reference_subject_images=image,
- ).images,
- "run successfully",
- )
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- core = ov.Core()
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=3, options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- compiled_unet = core.compile_model(unet_ir_xml_path, device.value)
- compiled_text_encoder = core.compile_model(text_encoder_ir_xml_path, device.value)
- compiled_image_encoder = core.compile_model(image_encoder_ir_xml_path, device.value)
- compiled_postfuse_module = core.compile_model(postfuse_module_ir_xml_path, device.value)
- compiled_object_transforms = core.compile_model(object_transforms_ir_xml_path, device.value)
-
- wrapped_model = ModelWrapper(
- create_pipeline(
- config,
- text_encoder=compiled_text_encoder,
- image_encoder=compiled_image_encoder,
- unet=compiled_unet,
- object_transforms=compiled_object_transforms,
- postfuse_module=compiled_postfuse_module,
- )
- )
-
-
-
-.. parsed-literal::
-
- Loading pipeline components...: 0%| | 0/7 [00:00`__
+
+ - `Install required packages <#install-required-packages>`__
+
+- `Checking NPU with Query Device <#checking-npu-with-query-device>`__
+
+ - `List the NPU with
+ core.available_devices <#list-the-npu-with-core-available_devices>`__
+ - `Check Properties with
+ core.get_property <#check-properties-with-core-get_property>`__
+ - `Brief Descriptions of Key
+ Properties <#brief-descriptions-of-key-properties>`__
+
+- `Compiling a Model on NPU <#compiling-a-model-on-npu>`__
+
+ - `Download and Convert a Model <#download-and-convert-a-model>`__
+
+ - `Download the Model <#download-the-model>`__
+ - `Convert the Model to OpenVINO IR
+ format <#convert-the-model-to-openvino-ir-format>`__
+
+ - `Compile with Default
+ Configuration <#compile-with-default-configuration>`__
+ - `Reduce Compile Time through Model
+ Caching <#reduce-compile-time-through-model-caching>`__
+
+ - `UMD Model Caching <#umd-model-caching>`__
+ - `OpenVINO Model Caching <#openvino-model-caching>`__
+
+ - `Throughput and Latency Performance
+ Hints <#throughput-and-latency-performance-hints>`__
+
+- `Performance Comparison with
+ benchmark_app <#performance-comparison-with-benchmark_app>`__
+
+ - `NPU vs CPU with Latency Hint <#npu-vs-cpu-with-latency-hint>`__
+
+ - `Effects of UMD Model
+ Caching <#effects-of-umd-model-caching>`__
+
+ - `NPU vs CPU with Throughput
+ Hint <#npu-vs-cpu-with-throughput-hint>`__
+
+- `Limitations <#limitations>`__
+- `Conclusion <#conclusion>`__
+
+This tutorial provides a high-level overview of working with the NPU
+device **Intel(R) AI Boost** (introduced with the Intel® Core™ Ultra
+generation of CPUs) in OpenVINO. It explains some of the key properties
+of the NPU and shows how to compile a model on NPU with performance
+hints.
+
+This tutorial also shows example commands for benchmark_app that can be
+run to compare NPU performance with CPU in different configurations.
+
+Introduction
+------------
+
+
+
+The Neural Processing Unit (NPU) is a low power hardware solution which
+enables you to offload certain neural network computation tasks from
+other devices, for more streamlined resource management.
+
+Note that the NPU plugin is included in PIP installation of OpenVINO™
+and you need to `install a proper NPU
+driver `__
+to use it successfully.
+
+| **Supported Platforms**:
+| Host: Intel® Core™ Ultra
+| NPU device: NPU 3720
+| OS: Ubuntu 22.04 (with Linux Kernel 6.6+), MS Windows 11 (both 64-bit)
+
+To learn more about the NPU Device, see the
+`page `__.
+
+Install required packages
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+.. code:: ipython3
+
+ %pip install -q "openvino>=2024.1.0" torch torchvision --extra-index-url https://download.pytorch.org/whl/cpu
+
+
+.. parsed-literal::
+
+ Note: you may need to restart the kernel to use updated packages.
+
+
+Checking NPU with Query Device
+------------------------------
+
+
+
+In this section, we will see how to list the available NPU and check its
+properties. Some of the key properties will be defined.
+
+List the NPU with core.available_devices
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+OpenVINO Runtime provides the ``available_devices`` method for checking
+which devices are available for inference. The following code will
+output a list a compatible OpenVINO devices, in which Intel NPU should
+appear (ensure that the driver is installed successfully).
+
+.. code:: ipython3
+
+ import openvino as ov
+
+ core = ov.Core()
+ core.available_devices
+
+
+
+
+.. parsed-literal::
+
+ ['CPU', 'GPU', 'NPU']
+
+
+
+Check Properties with core.get_property
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+To get information about the NPU, we can use device properties. In
+OpenVINO, devices have properties that describe their characteristics
+and configurations. Each property has a name and associated value that
+can be queried with the ``get_property`` method.
+
+To get the value of a property, such as the device name, we can use the
+``get_property`` method as follows:
+
+.. code:: ipython3
+
+ device = "NPU"
+
+ core.get_property(device, "FULL_DEVICE_NAME")
+
+
+
+
+.. parsed-literal::
+
+ 'Intel(R) AI Boost'
+
+
+
+Each device also has a specific property called
+``SUPPORTED_PROPERTIES``, that enables viewing all the available
+properties in the device. We can check the value for each property by
+simply looping through the dictionary returned by
+``core.get_property("NPU", "SUPPORTED_PROPERTIES")`` and then querying
+for that property.
+
+.. code:: ipython3
+
+ print(f"{device} SUPPORTED_PROPERTIES:\n")
+ supported_properties = core.get_property(device, "SUPPORTED_PROPERTIES")
+ indent = len(max(supported_properties, key=len))
+
+ for property_key in supported_properties:
+ if property_key not in ("SUPPORTED_METRICS", "SUPPORTED_CONFIG_KEYS", "SUPPORTED_PROPERTIES"):
+ try:
+ property_val = core.get_property(device, property_key)
+ except TypeError:
+ property_val = "UNSUPPORTED TYPE"
+ print(f"{property_key:<{indent}}: {property_val}")
+
+Brief Descriptions of Key Properties
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+Each device has several properties as seen in the last command. Some of
+the key properties are: - ``FULL_DEVICE_NAME`` - The product name of the
+NPU. - ``PERFORMANCE_HINT`` - A high-level way to tune the device for a
+specific performance metric, such as latency or throughput, without
+worrying about device-specific settings. - ``CACHE_DIR`` - The directory
+where the OpenVINO model cache data is stored to speed up the
+compilation time. - ``OPTIMIZATION_CAPABILITIES`` - The model data types
+(INT8, FP16, FP32, etc) that are supported by this NPU.
+
+To learn more about devices and properties, see the `Query Device
+Properties `__
+page.
+
+Compiling a Model on NPU
+------------------------
+
+
+
+Now, we know the NPU present in the system and we have checked its
+properties. We can easily use it for compiling and running models with
+OpenVINO NPU plugin.
+
+Download and Convert a Model
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+This tutorial uses the ``resnet50`` model. The ``resnet50`` model is
+used for image classification tasks. The model was trained on
+`ImageNet `__ dataset which
+contains over a million images categorized into 1000 classes. To read
+more about resnet50, see the
+`paper `__.
+
+Download the Model
+^^^^^^^^^^^^^^^^^^
+
+
+
+Fetch `ResNet50
+CV `__
+Classification model from torchvision.
+
+.. code:: ipython3
+
+ from pathlib import Path
+
+ # create a directory for resnet model file
+ MODEL_DIRECTORY_PATH = Path("model")
+ MODEL_DIRECTORY_PATH.mkdir(exist_ok=True)
+
+ model_name = "resnet50"
+
+.. code:: ipython3
+
+ from torchvision.models import resnet50, ResNet50_Weights
+
+ # create model object
+ pytorch_model = resnet50(weights=ResNet50_Weights.DEFAULT)
+
+ # switch model from training to inference mode
+ pytorch_model.eval();
+
+Convert the Model to OpenVINO IR format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+
+To convert this Pytorch model to OpenVINO IR with ``FP16`` precision,
+use model conversion API. The models are saved to the
+``model/ir_model/`` directory. For more details about model conversion,
+see this
+`page `__.
+
+.. code:: ipython3
+
+ precision = "FP16"
+
+ model_path = MODEL_DIRECTORY_PATH / "ir_model" / f"{model_name}_{precision.lower()}.xml"
+
+ model = None
+ if not model_path.exists():
+ model = ov.convert_model(pytorch_model, input=[[1, 3, 224, 224]])
+ ov.save_model(model, model_path, compress_to_fp16=(precision == "FP16"))
+ print("IR model saved to {}".format(model_path))
+ else:
+ print("Read IR model from {}".format(model_path))
+ model = core.read_model(model_path)
+
+
+.. parsed-literal::
+
+ Read IR model from model\ir_model\resnet50_fp16.xml
+
+
+**Note:** NPU also supports ``INT8`` quantized models.
+
+Compile with Default Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+When the model is ready, first we need to read it, using the
+``read_model`` method. Then, we can use the ``compile_model`` method and
+specify the name of the device we want to compile the model on, in this
+case, “NPU”.
+
+.. code:: ipython3
+
+ compiled_model = core.compile_model(model, device)
+
+Reduce Compile Time through Model Caching
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+Depending on the model used, device-specific optimizations and network
+compilations can cause the compile step to be time-consuming, especially
+with larger models, which may lead to bad user experience in the
+application. To solve this **Model Caching** can be used.
+
+Model Caching helps reduce application startup delays by exporting and
+reusing the compiled model automatically. The following two
+compilation-related metrics are crucial in this area:
+
+- **First-Ever Inference Latency (FEIL)**:
+ Measures all steps required to compile and execute a model on the
+ device for the first time. It includes model compilation time, the
+ time required to load and initialize the model on the device and the
+ first inference execution.
+- **First Inference Latency (FIL)**:
+ Measures the time required to load and initialize the pre-compiled
+ model on the device and the first inference execution.
+
+In NPU, UMD model caching is a solution enabled by default by the
+driver. It improves time to first inference (FIL) by storing the model
+in the cache after compilation (included in FEIL). Learn more about UMD
+Caching
+`here `__.
+Due to this caching, it takes lesser time to load the model after first
+compilation.
+
+| You can also use OpenVINO Model Caching, which is a common mechanism
+ for all OpenVINO device plugins and can be enabled by setting the
+ ``cache_dir`` property.
+| By enabling OpenVINO Model Caching, the UMD caching is automatically
+ bypassed by the NPU plugin, which means the model will only be stored
+ in the OpenVINO cache after compilation. When a cache hit occurs for
+ subsequent compilation requests, the plugin will import the model
+ instead of recompiling it.
+
+UMD Model Caching
+^^^^^^^^^^^^^^^^^
+
+
+
+To see how UMD caching see the following example:
+
+.. code:: ipython3
+
+ import time
+ from pathlib import Path
+
+ start = time.time()
+ core = ov.Core()
+
+ # Compile the model as before
+ model = core.read_model(model=model_path)
+ compiled_model = core.compile_model(model, device)
+ print(f"UMD Caching (first time) - compile time: {time.time() - start}s")
+
+
+.. parsed-literal::
+
+ UMD Caching (first time) - compile time: 3.2854952812194824s
+
+
+.. code:: ipython3
+
+ start = time.time()
+ core = ov.Core()
+
+ # Compile the model once again to see UMD Caching
+ model = core.read_model(model=model_path)
+ compiled_model = core.compile_model(model, device)
+ print(f"UMD Caching - compile time: {time.time() - start}s")
+
+
+.. parsed-literal::
+
+ UMD Caching - compile time: 2.269814968109131s
+
+
+OpenVINO Model Caching
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+
+To get an idea of OpenVINO model caching, we can use the OpenVINO cache
+as follow
+
+.. code:: ipython3
+
+ # Create cache folder
+ cache_folder = Path("cache")
+ cache_folder.mkdir(exist_ok=True)
+
+ start = time.time()
+ core = ov.Core()
+
+ # Set cache folder
+ core.set_property({"CACHE_DIR": cache_folder})
+
+ # Compile the model
+ model = core.read_model(model=model_path)
+ compiled_model = core.compile_model(model, device)
+ print(f"Cache enabled (first time) - compile time: {time.time() - start}s")
+
+ start = time.time()
+ core = ov.Core()
+
+ # Set cache folder
+ core.set_property({"CACHE_DIR": cache_folder})
+
+ # Compile the model as before
+ model = core.read_model(model=model_path)
+ compiled_model = core.compile_model(model, device)
+ print(f"Cache enabled (second time) - compile time: {time.time() - start}s")
+
+
+.. parsed-literal::
+
+ Cache enabled (first time) - compile time: 0.6362860202789307s
+ Cache enabled (second time) - compile time: 0.3032548427581787s
+
+
+And when the OpenVINO cache is disabled:
+
+.. code:: ipython3
+
+ start = time.time()
+ core = ov.Core()
+ model = core.read_model(model=model_path)
+ compiled_model = core.compile_model(model, device)
+ print(f"Cache disabled - compile time: {time.time() - start}s")
+
+
+.. parsed-literal::
+
+ Cache disabled - compile time: 3.0127954483032227s
+
+
+The actual time improvements will depend on the environment as well as
+the model being used but it is definitely something to consider when
+optimizing an application. To read more about this, see the `Model
+Caching
+docs `__.
+
+Throughput and Latency Performance Hints
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+To simplify device and pipeline configuration, OpenVINO provides
+high-level performance hints that automatically set the batch size and
+number of parallel threads for inference. The “LATENCY” performance hint
+optimizes for fast inference times while the “THROUGHPUT” performance
+hint optimizes for high overall bandwidth or FPS.
+
+To use the “LATENCY” performance hint, add
+``{"PERFORMANCE_HINT": "LATENCY"}`` when compiling the model as shown
+below. For NPU, this automatically minimizes the batch size and number
+of parallel streams such that all of the compute resources can focus on
+completing a single inference as fast as possible.
+
+.. code:: ipython3
+
+ compiled_model = core.compile_model(model, device, {"PERFORMANCE_HINT": "LATENCY"})
+
+To use the “THROUGHPUT” performance hint, add
+``{"PERFORMANCE_HINT": "THROUGHPUT"}`` when compiling the model. For
+NPUs, this creates multiple processing streams to efficiently utilize
+all the execution cores and optimizes the batch size to fill the
+available memory.
+
+.. code:: ipython3
+
+ compiled_model = core.compile_model(model, device, {"PERFORMANCE_HINT": "THROUGHPUT"})
+
+Performance Comparison with benchmark_app
+-----------------------------------------
+
+
+
+Given all the different options available when compiling a model, it may
+be difficult to know which settings work best for a certain application.
+Thankfully, OpenVINO provides ``benchmark_app`` - a performance
+benchmarking tool.
+
+The basic syntax of ``benchmark_app`` is as follows:
+
+``benchmark_app -m PATH_TO_MODEL -d TARGET_DEVICE -hint {throughput,cumulative_throughput,latency,none}``
+
+where ``TARGET_DEVICE`` is any device shown by the ``available_devices``
+method as well as the MULTI and AUTO devices we saw previously, and the
+value of hint should be one of the values between brackets.
+
+Note that benchmark_app only requires the model path to run but both
+device and hint arguments will be useful to us. For more advanced
+usages, the tool itself has other options that can be checked by running
+``benchmark_app -h`` or reading the
+`docs `__.
+The following example shows us to benchmark a simple model, using a NPU
+with latency focus:
+
+``benchmark_app -m {model_path} -d NPU -hint latency``
+
+| For completeness, let us list here some of the comparisons we may want
+ to do by varying the device and hint used. Note that the actual
+ performance may depend on the hardware used. Generally, we should
+ expect NPU to be better than CPU.
+| Please refer to the ``benchmark_app`` log entries under
+ ``[Step 11/11] Dumping statistics report`` to observe the differences
+ in latency and throughput between the CPU and NPU..
+
+NPU vs CPU with Latency Hint
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+
+.. code:: ipython3
+
+ !benchmark_app -m {model_path} -d CPU -hint latency
+
+
+.. parsed-literal::
+
+ [Step 1/11] Parsing and validating input arguments
+ [ INFO ] Parsing input parameters
+ [Step 2/11] Loading OpenVINO Runtime
+ [ INFO ] OpenVINO:
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ] Device info:
+ [ INFO ] CPU
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ]
+ [Step 3/11] Setting device configuration
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 14.00 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 143.22 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] NETWORK_NAME: Model2
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] NUM_STREAMS: 1
+ [ INFO ] AFFINITY: Affinity.HYBRID_AWARE
+ [ INFO ] INFERENCE_NUM_THREADS: 12
+ [ INFO ] PERF_COUNT: NO
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] PERFORMANCE_HINT: LATENCY
+ [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
+ [ INFO ] ENABLE_CPU_PINNING: False
+ [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
+ [ INFO ] ENABLE_HYPER_THREADING: False
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
+ [ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
+ [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
+ [ INFO ] KV_CACHE_PRECISION:
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
+ [ INFO ] Fill input 'x' with random values
+ [Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 28.95 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:['CPU']
+ [ INFO ] Count: 1612 iterations
+ [ INFO ] Duration: 60039.72 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 39.99 ms
+ [ INFO ] Average: 37.13 ms
+ [ INFO ] Min: 19.13 ms
+ [ INFO ] Max: 71.94 ms
+ [ INFO ] Throughput: 26.85 FPS
+
+
+.. code:: ipython3
+
+ !benchmark_app -m {model_path} -d NPU -hint latency
+
+
+.. parsed-literal::
+
+ [Step 1/11] Parsing and validating input arguments
+ [ INFO ] Parsing input parameters
+ [Step 2/11] Loading OpenVINO Runtime
+ [ INFO ] OpenVINO:
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ] Device info:
+ [ INFO ] NPU
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ]
+ [Step 3/11] Setting device configuration
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 11.51 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 2302.40 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] DEVICE_ID:
+ [ INFO ] ENABLE_CPU_PINNING: False
+ [ INFO ] EXECUTION_DEVICES: NPU.3720
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
+ [ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] NETWORK_NAME:
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
+ [ INFO ] PERF_COUNT: False
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
+ [ INFO ] Fill input 'x' with random values
+ [Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 7.94 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:NPU.3720
+ [ INFO ] Count: 17908 iterations
+ [ INFO ] Duration: 60004.49 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 3.29 ms
+ [ INFO ] Average: 3.33 ms
+ [ INFO ] Min: 3.21 ms
+ [ INFO ] Max: 6.90 ms
+ [ INFO ] Throughput: 298.44 FPS
+
+
+Effects of UMD Model Caching
+''''''''''''''''''''''''''''
+
+
+
+To see the effects of UMD Model caching, we are going to run the
+benchmark_app and see the difference in model read time and compilation
+time:
+
+.. code:: ipython3
+
+ !benchmark_app -m {model_path} -d NPU -hint latency
+
+
+.. parsed-literal::
+
+ [Step 1/11] Parsing and validating input arguments
+ [ INFO ] Parsing input parameters
+ [Step 2/11] Loading OpenVINO Runtime
+ [ INFO ] OpenVINO:
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ] Device info:
+ [ INFO ] NPU
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ]
+ [Step 3/11] Setting device configuration
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 11.00 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 2157.58 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] DEVICE_ID:
+ [ INFO ] ENABLE_CPU_PINNING: False
+ [ INFO ] EXECUTION_DEVICES: NPU.3720
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
+ [ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] NETWORK_NAME:
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
+ [ INFO ] PERF_COUNT: False
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
+ [ INFO ] Fill input 'x' with random values
+ [Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 60000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 7.94 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:NPU.3720
+ [ INFO ] Count: 17894 iterations
+ [ INFO ] Duration: 60004.76 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 3.29 ms
+ [ INFO ] Average: 3.33 ms
+ [ INFO ] Min: 3.21 ms
+ [ INFO ] Max: 14.38 ms
+ [ INFO ] Throughput: 298.21 FPS
+
+
+As you can see from the log entries ``[Step 4/11] Reading model files``
+and ``[Step 7/11] Loading the model to the device``, it takes less time
+to read and compile the model after the initial load.
+
+NPU vs CPU with Throughput Hint
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+
+.. code:: ipython3
+
+ !benchmark_app -m {model_path} -d CPU -hint throughput
+
+
+.. parsed-literal::
+
+ [Step 1/11] Parsing and validating input arguments
+ [ INFO ] Parsing input parameters
+ [Step 2/11] Loading OpenVINO Runtime
+ [ INFO ] OpenVINO:
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ] Device info:
+ [ INFO ] CPU
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ]
+ [Step 3/11] Setting device configuration
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 12.00 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 177.18 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] NETWORK_NAME: Model2
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
+ [ INFO ] NUM_STREAMS: 4
+ [ INFO ] AFFINITY: Affinity.HYBRID_AWARE
+ [ INFO ] INFERENCE_NUM_THREADS: 16
+ [ INFO ] PERF_COUNT: NO
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] PERFORMANCE_HINT: THROUGHPUT
+ [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
+ [ INFO ] ENABLE_CPU_PINNING: False
+ [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
+ [ INFO ] ENABLE_HYPER_THREADING: True
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
+ [ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
+ [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
+ [ INFO ] KV_CACHE_PRECISION:
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
+ [ INFO ] Fill input 'x' with random values
+ [Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 31.62 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:['CPU']
+ [ INFO ] Count: 3212 iterations
+ [ INFO ] Duration: 60082.26 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 65.28 ms
+ [ INFO ] Average: 74.60 ms
+ [ INFO ] Min: 35.65 ms
+ [ INFO ] Max: 157.31 ms
+ [ INFO ] Throughput: 53.46 FPS
+
+
+.. code:: ipython3
+
+ !benchmark_app -m {model_path} -d NPU -hint throughput
+
+
+.. parsed-literal::
+
+ [Step 1/11] Parsing and validating input arguments
+ [ INFO ] Parsing input parameters
+ [Step 2/11] Loading OpenVINO Runtime
+ [ INFO ] OpenVINO:
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ] Device info:
+ [ INFO ] NPU
+ [ INFO ] Build ................................. 2024.1.0-14992-621b025bef4
+ [ INFO ]
+ [ INFO ]
+ [Step 3/11] Setting device configuration
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 11.50 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : f32 / [...] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] x (node: x) : u8 / [N,C,H,W] / [1,3,224,224]
+ [ INFO ] Model outputs:
+ [ INFO ] x.45 (node: aten::linear/Add) : f32 / [...] / [1,1000]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 2265.07 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] DEVICE_ID:
+ [ INFO ] ENABLE_CPU_PINNING: False
+ [ INFO ] EXECUTION_DEVICES: NPU.3720
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] INTERNAL_SUPPORTED_PROPERTIES: {'CACHING_PROPERTIES': 'RO'}
+ [ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] NETWORK_NAME:
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
+ [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 1
+ [ INFO ] PERF_COUNT: False
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
+ [ INFO ] Fill input 'x' with random values
+ [Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 7.95 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:NPU.3720
+ [ INFO ] Count: 19080 iterations
+ [ INFO ] Duration: 60024.79 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 12.51 ms
+ [ INFO ] Average: 12.56 ms
+ [ INFO ] Min: 6.92 ms
+ [ INFO ] Max: 25.80 ms
+ [ INFO ] Throughput: 317.87 FPS
+
+
+Limitations
+-----------
+
+
+
+1. Currently, only the models with static shapes are supported on NPU.
+2. If the path to the model file includes non-Unicode symbols, such as
+ in Chinese, the model cannot be used for inference on NPU. It will
+ return an error.
+
+Conclusion
+----------
+
+
+
+This tutorial demonstrates how easy it is to use NPU in OpenVINO, check
+its properties, and even tailor the model performance through the
+different performance hints.
+
+Discover the power of Neural Processing Unit (NPU) with OpenVINO through
+these interactive Jupyter notebooks: ##### Introduction -
+`hello-world `__:
+Start your OpenVINO journey by performing inference on an OpenVINO IR
+model. -
+`hello-segmentation `__:
+Dive into inference with a segmentation model and explore image
+segmentation capabilities.
+
+Model Optimization and Conversion
+'''''''''''''''''''''''''''''''''
+
+- `model-tools `__:
+ Discover how to download, convert, and benchmark models from the Open
+ Model Zoo.
+- `tflite-to-openvino `__:
+ Learn the process of converting TensorFlow Lite models to OpenVINO IR
+ format.
+- `yolov7-optimization `__:
+ Optimize the YOLOv7 model for enhanced performance in OpenVINO.
+- `yolov8-optimization `__:
+ Convert and optimize YOLOv8 models for efficient deployment with
+ OpenVINO.
+
+Advanced Computer Vision Techniques
+'''''''''''''''''''''''''''''''''''
+
+- `vision-background-removal `__:
+ Implement advanced image segmentation and background manipulation
+ with U^2-Net.
+- `handwritten-ocr `__:
+ Apply optical character recognition to handwritten Chinese and
+ Japanese text.
+- `image-inpainting `__:
+ Explore the art of image in-painting and restore images with missing
+ parts.
+- `vehicle-detection-and-recognition `__:
+ Use pre-trained models for vehicle detection and recognition in
+ images.
+- `vision-image-colorization `__:
+ Bring black and white images to life by adding color with neural
+ networks.
+
+Real-Time Webcam Applications
+'''''''''''''''''''''''''''''
+
+- `tflite-selfie-segmentation `__:
+ Apply TensorFlow Lite models for selfie segmentation and background
+ processing.
+- `object-detection-webcam `__:
+ Experience real-time object detection using your webcam and OpenVINO.
+- `pose-estimation-webcam `__:
+ Perform human pose estimation in real-time with webcam integration.
+- `action-recognition-webcam `__:
+ Recognize and classify human actions live with your webcam.
+- `style-transfer-webcam `__:
+ Transform your webcam feed with artistic styles in real-time using
+ pre-trained models.
+- `3D-pose-estimation-webcam `__:
+ Perform 3D multi-person pose estimation with OpenVINO.
diff --git a/docs/notebooks/hello-segmentation-with-output.rst b/docs/notebooks/hello-segmentation-with-output.rst
index 8eb9901014a..6ccbe6efafa 100644
--- a/docs/notebooks/hello-segmentation-with-output.rst
+++ b/docs/notebooks/hello-segmentation-with-output.rst
@@ -38,10 +38,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -198,7 +194,7 @@ is provided.
.. parsed-literal::
-
+
@@ -225,7 +221,7 @@ Do Inference
.. parsed-literal::
-
+
diff --git a/docs/notebooks/hello-world-with-output.rst b/docs/notebooks/hello-world-with-output.rst
index ea0f5032f44..b2cc053a7bd 100644
--- a/docs/notebooks/hello-world-with-output.rst
+++ b/docs/notebooks/hello-world-with-output.rst
@@ -42,10 +42,6 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
diff --git a/docs/notebooks/hugging-face-hub-with-output.rst b/docs/notebooks/hugging-face-hub-with-output.rst
index e7ed9c3e9a7..8a049bb0612 100644
--- a/docs/notebooks/hugging-face-hub-with-output.rst
+++ b/docs/notebooks/hugging-face-hub-with-output.rst
@@ -73,15 +73,7 @@ Installing Requirements
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -129,6 +121,8 @@ tutorials `__.
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-
-
-.. parsed-literal::
-
- 2024-04-17 23:58:07.506496: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:58:07.542304: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:16:16.370934: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:16:16.406467: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:58:08.114190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
+ 2024-05-07 00:16:17.026247: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
@@ -384,28 +366,14 @@ inference run.
.. parsed-literal::
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Framework not specified. Using pt to export the model.
-
-
-.. parsed-literal::
-
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
-
-
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
-
-
-.. parsed-literal::
-
+ Using framework PyTorch: 2.3.0+cpu
Overriding 1 configuration item(s)
-
-
-.. parsed-literal::
-
- use_cache -> False
@@ -416,12 +384,8 @@ inference run.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
-
-
-.. parsed-literal::
-
Compiling the model to AUTO ...
@@ -476,11 +440,7 @@ Full list of supported arguments available via ``--help``
.. parsed-literal::
- 2024-04-17 23:58:19.217675: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
+ 2024-05-07 00:16:28.374572: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
usage: optimum-cli export openvino [-h] -m MODEL [--task TASK]
[--cache_dir CACHE_DIR]
[--framework {pt,tf}] [--trust-remote-code]
@@ -489,7 +449,9 @@ Full list of supported arguments available via ``--help``
[--weight-format {fp32,fp16,int8,int4,int4_sym_g128,int4_asym_g128,int4_sym_g64,int4_asym_g64}]
[--ratio RATIO] [--sym]
[--group-size GROUP_SIZE]
- [--disable-stateful] [--convert-tokenizer]
+ [--dataset DATASET] [--disable-stateful]
+ [--disable-convert-tokenizer]
+ [--convert-tokenizer]
[--library {transformers,diffusers,timm,sentence_transformers}]
output
@@ -507,21 +469,21 @@ Full list of supported arguments available via ``--help``
--task TASK The task to export the model for. If not specified,
the task will be auto-inferred based on the model.
Available tasks depend on the model, but are among:
- ['masked-im', 'fill-mask', 'token-classification',
- 'conversational', 'image-segmentation', 'zero-shot-
- image-classification', 'audio-frame-classification',
- 'audio-classification', 'feature-extraction', 'zero-
- shot-object-detection', 'text2text-generation',
- 'automatic-speech-recognition', 'text-generation',
- 'image-to-text', 'semantic-segmentation', 'text-
- classification', 'sentence-similarity', 'audio-
- xvector', 'depth-estimation', 'object-detection',
- 'stable-diffusion', 'image-classification', 'mask-
- generation', 'multiple-choice', 'stable-diffusion-xl',
- 'image-to-image', 'text-to-audio', 'question-
- answering']. For decoder models, use `xxx-with-past`
- to export the model using past key values in the
- decoder.
+ ['feature-extraction', 'zero-shot-image-
+ classification', 'conversational', 'text2text-
+ generation', 'sentence-similarity', 'stable-
+ diffusion', 'text-to-audio', 'stable-diffusion-xl',
+ 'fill-mask', 'image-to-text', 'text-generation',
+ 'semantic-segmentation', 'automatic-speech-
+ recognition', 'mask-generation', 'token-
+ classification', 'audio-classification', 'multiple-
+ choice', 'question-answering', 'masked-im', 'zero-
+ shot-object-detection', 'audio-xvector', 'image-
+ segmentation', 'object-detection', 'image-
+ classification', 'audio-frame-classification', 'image-
+ to-image', 'depth-estimation', 'text-classification'].
+ For decoder models, use `xxx-with-past` to export the
+ model using past key values in the decoder.
--cache_dir CACHE_DIR
Path indicating where to store cache.
--framework {pt,tf} The framework to use for the export. If not provided,
@@ -552,6 +514,13 @@ Full list of supported arguments available via ``--help``
--group-size GROUP_SIZE
The group size to use for quantization. Recommended
value is 128 and -1 uses per-column quantization.
+ --dataset DATASET The dataset used for data-aware compression or
+ quantization with NNCF. You can use the one from the
+ list ['wikitext2','c4','c4-new','ptb','ptb-new'] for
+ LLLMs or
+ ['conceptual_captions','laion/220k-GPT4Vision-
+ captions-from-LIVIS','laion/filtered-wit'] for
+ diffusion models.
--disable-stateful Disable stateful converted models, stateless models
will be generated instead. Stateful models are
produced by default when this key is not used. In
@@ -563,8 +532,11 @@ Full list of supported arguments available via ``--help``
a stateless model, for example, to be compatible with
existing OpenVINO native inference code that expects
kv-cache inputs and outputs in the model.
- --convert-tokenizer Add converted tokenizer and detokenizer with OpenVINO
- Tokenizers
+ --disable-convert-tokenizer
+ Do not add converted tokenizer and detokenizer
+ OpenVINO models.
+ --convert-tokenizer [Deprecated] Add converted tokenizer and detokenizer
+ with OpenVINO Tokenizers.
--library {transformers,diffusers,timm,sentence_transformers}
The library on the model. If not provided, will
attempt to infer the local checkpoint's library
@@ -588,44 +560,25 @@ compression:
.. parsed-literal::
- 2024-04-17 23:58:23.821792: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
+ 2024-05-07 00:16:33.077270: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
-
-
-.. parsed-literal::
-
`--fp16` option is deprecated and will be removed in a future version. Use `--weight-format` instead.
-
-
-.. parsed-literal::
-
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Framework not specified. Using pt to export the model.
-
-
-.. parsed-literal::
-
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
-
-
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
+ Using framework PyTorch: 2.3.0+cpu
Overriding 1 configuration item(s)
- use_cache -> False
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
+ OpenVINO Tokenizers is not available. To deploy models in production with C++ code, please follow installation instructions: https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#installation
+
+ Tokenizer won't be converted.
After export, model will be available in the specified directory and can
diff --git a/docs/notebooks/image-classification-quantization-with-output.rst b/docs/notebooks/image-classification-quantization-with-output.rst
index e4d56582cdf..47d758967f2 100644
--- a/docs/notebooks/image-classification-quantization-with-output.rst
+++ b/docs/notebooks/image-classification-quantization-with-output.rst
@@ -44,10 +44,10 @@ Table of contents:
.. code:: ipython3
import platform
-
+
# Install required packages
%pip install -q "openvino>=2023.1.0" "nncf>=2.6.0" torch torchvision tqdm --extra-index-url https://download.pytorch.org/whl/cpu
-
+
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
@@ -57,22 +57,18 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
.. code:: ipython3
from pathlib import Path
-
+
# Set the data and model directories
DATA_DIR = Path("data")
MODEL_DIR = Path("model")
model_repo = "pytorch-cifar-models"
-
+
DATA_DIR.mkdir(exist_ok=True)
MODEL_DIR.mkdir(exist_ok=True)
@@ -91,383 +87,28 @@ Model preparation stage has the following steps:
.. code:: ipython3
import sys
-
+
if not Path(model_repo).exists():
!git clone https://github.com/chenyaofo/pytorch-cifar-models.git
-
+
sys.path.append(model_repo)
.. parsed-literal::
Cloning into 'pytorch-cifar-models'...
-
-
-.. parsed-literal::
-
remote: Enumerating objects: 282, done.[K
- remote: Counting objects: 0% (1/281)[K
-remote: Counting objects: 1% (3/281)[K
-remote: Counting objects: 2% (6/281)[K
-remote: Counting objects: 3% (9/281)[K
-remote: Counting objects: 4% (12/281)[K
-remote: Counting objects: 5% (15/281)[K
-remote: Counting objects: 6% (17/281)[K
-remote: Counting objects: 7% (20/281)[K
-remote: Counting objects: 8% (23/281)[K
-remote: Counting objects: 9% (26/281)[K
-remote: Counting objects: 10% (29/281)[K
-remote: Counting objects: 11% (31/281)[K
-remote: Counting objects: 12% (34/281)[K
-remote: Counting objects: 13% (37/281)[K
-remote: Counting objects: 14% (40/281)[K
-remote: Counting objects: 15% (43/281)[K
-remote: Counting objects: 16% (45/281)[K
-remote: Counting objects: 17% (48/281)[K
-remote: Counting objects: 18% (51/281)[K
-remote: Counting objects: 19% (54/281)[K
-remote: Counting objects: 20% (57/281)[K
-remote: Counting objects: 21% (60/281)[K
-remote: Counting objects: 22% (62/281)[K
-remote: Counting objects: 23% (65/281)[K
-remote: Counting objects: 24% (68/281)[K
-remote: Counting objects: 25% (71/281)[K
-remote: Counting objects: 26% (74/281)[K
-remote: Counting objects: 27% (76/281)[K
-remote: Counting objects: 28% (79/281)[K
-remote: Counting objects: 29% (82/281)[K
-remote: Counting objects: 30% (85/281)[K
-remote: Counting objects: 31% (88/281)[K
-remote: Counting objects: 32% (90/281)[K
-remote: Counting objects: 33% (93/281)[K
-remote: Counting objects: 34% (96/281)[K
-remote: Counting objects: 35% (99/281)[K
-remote: Counting objects: 36% (102/281)[K
-remote: Counting objects: 37% (104/281)[K
-remote: Counting objects: 38% (107/281)[K
-remote: Counting objects: 39% (110/281)[K
-remote: Counting objects: 40% (113/281)[K
-remote: Counting objects: 41% (116/281)[K
-remote: Counting objects: 42% (119/281)[K
-remote: Counting objects: 43% (121/281)[K
-remote: Counting objects: 44% (124/281)[K
-remote: Counting objects: 45% (127/281)[K
-remote: Counting objects: 46% (130/281)[K
-remote: Counting objects: 47% (133/281)[K
-remote: Counting objects: 48% (135/281)[K
-remote: Counting objects: 49% (138/281)[K
-remote: Counting objects: 50% (141/281)[K
-remote: Counting objects: 51% (144/281)[K
-remote: Counting objects: 52% (147/281)[K
-remote: Counting objects: 53% (149/281)[K
-remote: Counting objects: 54% (152/281)[K
-remote: Counting objects: 55% (155/281)[K
-remote: Counting objects: 56% (158/281)[K
-remote: Counting objects: 57% (161/281)[K
-remote: Counting objects: 58% (163/281)[K
-remote: Counting objects: 59% (166/281)[K
-remote: Counting objects: 60% (169/281)[K
-remote: Counting objects: 61% (172/281)[K
-remote: Counting objects: 62% (175/281)[K
-remote: Counting objects: 63% (178/281)[K
-remote: Counting objects: 64% (180/281)[K
-remote: Counting objects: 65% (183/281)[K
-remote: Counting objects: 66% (186/281)[K
-remote: Counting objects: 67% (189/281)[K
-remote: Counting objects: 68% (192/281)[K
-remote: Counting objects: 69% (194/281)[K
-remote: Counting objects: 70% (197/281)[K
-remote: Counting objects: 71% (200/281)[K
-remote: Counting objects: 72% (203/281)[K
-remote: Counting objects: 73% (206/281)[K
-remote: Counting objects: 74% (208/281)[K
-remote: Counting objects: 75% (211/281)[K
-remote: Counting objects: 76% (214/281)[K
-remote: Counting objects: 77% (217/281)[K
-remote: Counting objects: 78% (220/281)[K
-remote: Counting objects: 79% (222/281)[K
-remote: Counting objects: 80% (225/281)[K
-remote: Counting objects: 81% (228/281)[K
-remote: Counting objects: 82% (231/281)[K
-remote: Counting objects: 83% (234/281)[K
-remote: Counting objects: 84% (237/281)[K
-remote: Counting objects: 85% (239/281)[K
-remote: Counting objects: 86% (242/281)[K
-remote: Counting objects: 87% (245/281)[K
-remote: Counting objects: 88% (248/281)[K
-remote: Counting objects: 89% (251/281)[K
-remote: Counting objects: 90% (253/281)[K
-remote: Counting objects: 91% (256/281)[K
-remote: Counting objects: 92% (259/281)[K
-remote: Counting objects: 93% (262/281)[K
-remote: Counting objects: 94% (265/281)[K
-remote: Counting objects: 95% (267/281)[K
-remote: Counting objects: 96% (270/281)[K
-remote: Counting objects: 97% (273/281)[K
-remote: Counting objects: 98% (276/281)[K
-remote: Counting objects: 99% (279/281)[K
-remote: Counting objects: 100% (281/281)[K
-remote: Counting objects: 100% (281/281), done.[K
- remote: Compressing objects: 1% (1/96)[K
-remote: Compressing objects: 2% (2/96)[K
-remote: Compressing objects: 3% (3/96)[K
-remote: Compressing objects: 4% (4/96)[K
-remote: Compressing objects: 5% (5/96)[K
-remote: Compressing objects: 6% (6/96)[K
-remote: Compressing objects: 7% (7/96)[K
-remote: Compressing objects: 8% (8/96)[K
-remote: Compressing objects: 9% (9/96)[K
-remote: Compressing objects: 10% (10/96)[K
-remote: Compressing objects: 11% (11/96)[K
-remote: Compressing objects: 12% (12/96)[K
-remote: Compressing objects: 13% (13/96)[K
-remote: Compressing objects: 14% (14/96)[K
-remote: Compressing objects: 15% (15/96)[K
-remote: Compressing objects: 16% (16/96)[K
-remote: Compressing objects: 17% (17/96)[K
-remote: Compressing objects: 18% (18/96)[K
-remote: Compressing objects: 19% (19/96)[K
-remote: Compressing objects: 20% (20/96)[K
-remote: Compressing objects: 21% (21/96)[K
-remote: Compressing objects: 22% (22/96)[K
-remote: Compressing objects: 23% (23/96)[K
-remote: Compressing objects: 25% (24/96)[K
-remote: Compressing objects: 26% (25/96)[K
-remote: Compressing objects: 27% (26/96)[K
-remote: Compressing objects: 28% (27/96)[K
-remote: Compressing objects: 29% (28/96)[K
-remote: Compressing objects: 30% (29/96)[K
-remote: Compressing objects: 31% (30/96)[K
-remote: Compressing objects: 32% (31/96)[K
-remote: Compressing objects: 33% (32/96)[K
-remote: Compressing objects: 34% (33/96)[K
-remote: Compressing objects: 35% (34/96)[K
-remote: Compressing objects: 36% (35/96)[K
-remote: Compressing objects: 37% (36/96)[K
-remote: Compressing objects: 38% (37/96)[K
-remote: Compressing objects: 39% (38/96)[K
-remote: Compressing objects: 40% (39/96)[K
-remote: Compressing objects: 41% (40/96)[K
-remote: Compressing objects: 42% (41/96)[K
-remote: Compressing objects: 43% (42/96)[K
-remote: Compressing objects: 44% (43/96)[K
-remote: Compressing objects: 45% (44/96)[K
-remote: Compressing objects: 46% (45/96)[K
-remote: Compressing objects: 47% (46/96)[K
-remote: Compressing objects: 48% (47/96)[K
-remote: Compressing objects: 50% (48/96)[K
-remote: Compressing objects: 51% (49/96)[K
-remote: Compressing objects: 52% (50/96)[K
-remote: Compressing objects: 53% (51/96)[K
-remote: Compressing objects: 54% (52/96)[K
-remote: Compressing objects: 55% (53/96)[K
-remote: Compressing objects: 56% (54/96)[K
-remote: Compressing objects: 57% (55/96)[K
-remote: Compressing objects: 58% (56/96)[K
-remote: Compressing objects: 59% (57/96)[K
-remote: Compressing objects: 60% (58/96)[K
-remote: Compressing objects: 61% (59/96)[K
-remote: Compressing objects: 62% (60/96)[K
-remote: Compressing objects: 63% (61/96)[K
-remote: Compressing objects: 64% (62/96)[K
-remote: Compressing objects: 65% (63/96)[K
-remote: Compressing objects: 66% (64/96)[K
-remote: Compressing objects: 67% (65/96)[K
-remote: Compressing objects: 68% (66/96)[K
-remote: Compressing objects: 69% (67/96)[K
-remote: Compressing objects: 70% (68/96)[K
-remote: Compressing objects: 71% (69/96)[K
-remote: Compressing objects: 72% (70/96)[K
-remote: Compressing objects: 73% (71/96)[K
-remote: Compressing objects: 75% (72/96)[K
-remote: Compressing objects: 76% (73/96)[K
-remote: Compressing objects: 77% (74/96)[K
-remote: Compressing objects: 78% (75/96)[K
-remote: Compressing objects: 79% (76/96)[K
-remote: Compressing objects: 80% (77/96)[K
-remote: Compressing objects: 81% (78/96)[K
-remote: Compressing objects: 82% (79/96)[K
-remote: Compressing objects: 83% (80/96)[K
-remote: Compressing objects: 84% (81/96)[K
-remote: Compressing objects: 85% (82/96)[K
-remote: Compressing objects: 86% (83/96)[K
-remote: Compressing objects: 87% (84/96)[K
-remote: Compressing objects: 88% (85/96)[K
-remote: Compressing objects: 89% (86/96)[K
-remote: Compressing objects: 90% (87/96)[K
-remote: Compressing objects: 91% (88/96)[K
-remote: Compressing objects: 92% (89/96)[K
-remote: Compressing objects: 93% (90/96)[K
-remote: Compressing objects: 94% (91/96)[K
-remote: Compressing objects: 95% (92/96)[K
-remote: Compressing objects: 96% (93/96)[K
-remote: Compressing objects: 97% (94/96)[K
-remote: Compressing objects: 98% (95/96)[K
-remote: Compressing objects: 100% (96/96)[K
-remote: Compressing objects: 100% (96/96), done.[K
-
-
-.. parsed-literal::
-
- Receiving objects: 0% (1/282)
-Receiving objects: 1% (3/282)
-Receiving objects: 2% (6/282)
-Receiving objects: 3% (9/282)
-Receiving objects: 4% (12/282)
-Receiving objects: 5% (15/282)
-Receiving objects: 6% (17/282)
-Receiving objects: 7% (20/282)
-Receiving objects: 8% (23/282)
-Receiving objects: 9% (26/282)
-Receiving objects: 10% (29/282)
-Receiving objects: 11% (32/282)
-Receiving objects: 12% (34/282)
-Receiving objects: 13% (37/282)
-Receiving objects: 14% (40/282)
-Receiving objects: 15% (43/282)
-Receiving objects: 16% (46/282)
-Receiving objects: 17% (48/282)
-Receiving objects: 18% (51/282)
-Receiving objects: 19% (54/282)
-Receiving objects: 20% (57/282)
-Receiving objects: 21% (60/282)
-Receiving objects: 22% (63/282)
-Receiving objects: 23% (65/282)
-Receiving objects: 24% (68/282)
-Receiving objects: 25% (71/282)
-Receiving objects: 26% (74/282)
-Receiving objects: 27% (77/282)
-Receiving objects: 28% (79/282)
-Receiving objects: 29% (82/282)
-Receiving objects: 30% (85/282)
-Receiving objects: 31% (88/282)
-Receiving objects: 32% (91/282)
-Receiving objects: 33% (94/282)
-Receiving objects: 34% (96/282)
-Receiving objects: 35% (99/282)
-Receiving objects: 36% (102/282)
-Receiving objects: 37% (105/282)
-Receiving objects: 38% (108/282)
-Receiving objects: 39% (110/282)
-Receiving objects: 40% (113/282)
-Receiving objects: 41% (116/282)
-Receiving objects: 42% (119/282)
-Receiving objects: 43% (122/282)
-Receiving objects: 44% (125/282)
-Receiving objects: 45% (127/282)
-Receiving objects: 46% (130/282)
-Receiving objects: 47% (133/282)
-Receiving objects: 48% (136/282)
-Receiving objects: 49% (139/282)
-Receiving objects: 50% (141/282)
-Receiving objects: 51% (144/282)
-Receiving objects: 52% (147/282)
-Receiving objects: 53% (150/282)
-Receiving objects: 54% (153/282)
-Receiving objects: 55% (156/282)
-Receiving objects: 56% (158/282)
-Receiving objects: 57% (161/282)
-Receiving objects: 58% (164/282)
-Receiving objects: 59% (167/282)
-Receiving objects: 60% (170/282)
-Receiving objects: 61% (173/282)
-Receiving objects: 62% (175/282)
-Receiving objects: 63% (178/282)
-Receiving objects: 64% (181/282)
-Receiving objects: 65% (184/282)
-Receiving objects: 66% (187/282)
-Receiving objects: 67% (189/282)
-Receiving objects: 68% (192/282)
-Receiving objects: 69% (195/282)
-Receiving objects: 70% (198/282)
-Receiving objects: 71% (201/282)
-Receiving objects: 72% (204/282)
-Receiving objects: 73% (206/282)
-Receiving objects: 74% (209/282)
-Receiving objects: 75% (212/282)
-
-.. parsed-literal::
-
- Receiving objects: 76% (215/282)
-
-.. parsed-literal::
-
- Receiving objects: 77% (218/282)
-Receiving objects: 78% (220/282)
-Receiving objects: 79% (223/282)
-Receiving objects: 80% (226/282)
-
-.. parsed-literal::
-
- Receiving objects: 81% (229/282)
-Receiving objects: 82% (232/282)
-Receiving objects: 83% (235/282)
-
-.. parsed-literal::
-
- Receiving objects: 84% (237/282)
-Receiving objects: 85% (240/282)
-Receiving objects: 86% (243/282)
-
-.. parsed-literal::
-
- Receiving objects: 87% (246/282)
-
-.. parsed-literal::
-
- Receiving objects: 88% (249/282)
-Receiving objects: 89% (251/282)
-remote: Total 282 (delta 135), reused 269 (delta 128), pack-reused 1[K
- Receiving objects: 90% (254/282)
-Receiving objects: 91% (257/282)
-Receiving objects: 92% (260/282)
-Receiving objects: 93% (263/282)
-Receiving objects: 94% (266/282)
-Receiving objects: 95% (268/282)
-Receiving objects: 96% (271/282)
-Receiving objects: 97% (274/282)
-Receiving objects: 98% (277/282)
-Receiving objects: 99% (280/282)
-Receiving objects: 100% (282/282)
-Receiving objects: 100% (282/282), 9.22 MiB | 21.40 MiB/s, done.
- Resolving deltas: 0% (0/135)
-Resolving deltas: 2% (4/135)
-Resolving deltas: 4% (6/135)
-Resolving deltas: 5% (7/135)
-Resolving deltas: 14% (19/135)
-Resolving deltas: 19% (26/135)
-Resolving deltas: 20% (27/135)
-Resolving deltas: 22% (31/135)
-Resolving deltas: 25% (34/135)
-Resolving deltas: 27% (37/135)
-Resolving deltas: 28% (38/135)
-Resolving deltas: 29% (40/135)
-Resolving deltas: 30% (41/135)
-Resolving deltas: 31% (42/135)
-Resolving deltas: 32% (44/135)
-Resolving deltas: 34% (47/135)
-Resolving deltas: 40% (54/135)
-Resolving deltas: 45% (62/135)
-Resolving deltas: 46% (63/135)
-Resolving deltas: 51% (69/135)
-Resolving deltas: 57% (78/135)
-Resolving deltas: 58% (79/135)
-Resolving deltas: 59% (80/135)
-Resolving deltas: 60% (82/135)
-Resolving deltas: 61% (83/135)
-Resolving deltas: 69% (94/135)
-Resolving deltas: 71% (97/135)
-
-.. parsed-literal::
-
- Resolving deltas: 100% (135/135)
-Resolving deltas: 100% (135/135), done.
+ remote: Counting objects: 100% (281/281), done.[K
+ remote: Compressing objects: 100% (96/96), done.[K
+ remote: Total 282 (delta 135), reused 269 (delta 128), pack-reused 1[K
+ Receiving objects: 100% (282/282), 9.22 MiB | 24.51 MiB/s, done.
+ Resolving deltas: 100% (135/135), done.
.. code:: ipython3
from pytorch_cifar_models import cifar10_mobilenetv2_x1_0
-
+
model = cifar10_mobilenetv2_x1_0(pretrained=True)
OpenVINO supports PyTorch models via conversion to OpenVINO Intermediate
@@ -485,11 +126,11 @@ can be found on this
.. code:: ipython3
import openvino as ov
-
+
model.eval()
-
+
ov_model = ov.convert_model(model, input=[1, 3, 32, 32])
-
+
ov.save_model(ov_model, MODEL_DIR / "mobilenet_v2.xml")
Prepare Dataset
@@ -508,7 +149,7 @@ Preprocessing for model obtained from training
import torch
from torchvision import transforms
from torchvision.datasets import CIFAR10
-
+
transform = transforms.Compose(
[
transforms.ToTensor(),
@@ -532,332 +173,7 @@ Preprocessing for model obtained from training
.. parsed-literal::
-
- 0%| | 0/170498071 [00:00
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model2
[ INFO ] NUM_STREAMS: 12
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12
@@ -1171,26 +476,23 @@ Tool
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model2
[ INFO ] NUM_STREAMS: 12
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12
@@ -1268,26 +563,23 @@ Tool `__ to
speedup the generation process. Previously, we already considered how to
convert and run SDXL model for Text-to-Image and Image-to-Image
generation using Optimum-Intel library (please check out this notebook
-for `details `__ ), now
+for `details `__), now
we will use it in combination with ControlNet and convert it using
OpenVINO Model Conversion API.
diff --git a/docs/notebooks/knowledge-graphs-conve-with-output.rst b/docs/notebooks/knowledge-graphs-conve-with-output.rst
index 76b9841d31e..225f8d7beba 100644
--- a/docs/notebooks/knowledge-graphs-conve-with-output.rst
+++ b/docs/notebooks/knowledge-graphs-conve-with-output.rst
@@ -222,7 +222,7 @@ Download Model Checkpoint
.. parsed-literal::
- PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/knowledge-graphs-conve/models/conve.pt')
+ PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/knowledge-graphs-conve/models/conve.pt')
@@ -384,7 +384,7 @@ typical to use metrics such as Mean Reciprocal Rank, Hits@10 etc.
.. parsed-literal::
- Average time taken for inference: 0.6107787291208903 ms
+ Average time taken for inference: 0.7582604885101318 ms
Mean accuracy of the model on the test dataset: 0.875
@@ -531,7 +531,7 @@ select device from dropdown list for running inference using OpenVINO
.. parsed-literal::
- Average time taken for inference: 0.6598830223083496 ms
+ Average time taken for inference: 0.675062338511149 ms
Mean accuracy of the model on the test dataset: 0.10416666666666667
@@ -550,7 +550,7 @@ Determine the platform specific speedup obtained through OpenVINO graph optimiza
.. parsed-literal::
- Speedup with OpenVINO optimizations: 0.93 X
+ Speedup with OpenVINO optimizations: 1.12 X
Benchmark the converted OpenVINO model using benchmark app
@@ -580,30 +580,22 @@ inference can also be obtained by looking at the benchmark app results.
.. parsed-literal::
Benchmark OpenVINO model using the benchmark app
-
-
-.. parsed-literal::
-
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 13.96 ms
+ [ INFO ] Read model took 4.87 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] e1 (node: e1) : i64 / [...] / []
@@ -619,11 +611,7 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] Model outputs:
[ INFO ] ***NO_NAME*** (node: aten::softmax/Softmax) : f32 / [...] / [1,271]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 82.08 ms
+ [ INFO ] Compile model took 69.30 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@@ -638,6 +626,7 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
@@ -652,21 +641,17 @@ inference can also be obtained by looking at the benchmark app results.
[ INFO ] Fill input 'rel' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 10000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
- [ INFO ] First inference took 1.43 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 1.25 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 101412 iterations
- [ INFO ] Duration: 10001.13 ms
+ [ INFO ] Count: 101688 iterations
+ [ INFO ] Duration: 10000.88 ms
[ INFO ] Latency:
[ INFO ] Median: 1.01 ms
[ INFO ] Average: 1.02 ms
- [ INFO ] Min: 0.60 ms
- [ INFO ] Max: 8.57 ms
- [ INFO ] Throughput: 10140.06 FPS
+ [ INFO ] Min: 0.70 ms
+ [ INFO ] Max: 8.75 ms
+ [ INFO ] Throughput: 10167.91 FPS
Conclusions
diff --git a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output.rst b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output.rst
index 8a39325ae71..174b518b1db 100644
--- a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output.rst
+++ b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output.rst
@@ -66,31 +66,10 @@ Install requirements
.. parsed-literal::
- Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
-
-
-.. parsed-literal::
-
+ Requirement already satisfied: pip in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (24.0)
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -154,18 +133,14 @@ example `__
.. parsed-literal::
- 2024-04-18 00:01:02.253505: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:01:02.287838: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:19:14.165808: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:19:14.200484: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:01:02.846922: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
+ 2024-05-07 00:19:14.695568: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
@@ -385,19 +360,11 @@ Vision model accept ``pixel_values`` and returns ``image_embeds``.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:471: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:469: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:511: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:509: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
@@ -425,7 +392,7 @@ Convert Image To Text Projection model
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:165: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torch/jit/_trace.py:165: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)
if a.grad is not None:
@@ -560,17 +527,13 @@ generated text by ``AutoProcessor``.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:810: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:808: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if max_pos > self.weights.size(0):
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1119: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1117: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if input_shape[-1] > 1:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:926: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:924: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attention_mask.size() != (batch_size, 1, seq_length, src_len):
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1212: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/kosmos2/modeling_kosmos2.py:1210: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if past_key_values_length > 0:
diff --git a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.jpg b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.jpg
index b400ee72fba..7117295e2da 100644
--- a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.jpg
+++ b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:c0a83eb915195973a7315333a239ea60abf597988effc7fcca7dfea302afb625
-size 117636
+oid sha256:9ab062bc4b0b22c2815ec0f5c3a2e7237c9ae71ac83cc94e3363de499d8a03c6
+size 118458
diff --git a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.png b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.png
index d0b339f53bf..50e6ce19cdb 100644
--- a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.png
+++ b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_29_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:8d18af510a639c26185fec562aed50889b07e16f73465e7cd071b97634995d78
-size 1151036
+oid sha256:a3943e3c5bcdf600ff01c05b9f60dc88b9bc7bd38474d51c247c4e80544d5964
+size 1150936
diff --git a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.jpg b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.jpg
index 4208969cbb7..d894c63b7d0 100644
--- a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.jpg
+++ b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:e9aca69f7a8aba308eed0932bb52ca703329c167786b855c33392c196775e3c7
-size 116335
+oid sha256:0541792d494f341f9b36f073472c0f6bc248297506ada281f7dfbb3feaf0a5af
+size 120137
diff --git a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.png b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.png
index 4ee528c804a..58b57b9d252 100644
--- a/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.png
+++ b/docs/notebooks/kosmos2-multimodal-large-language-model-with-output_files/kosmos2-multimodal-large-language-model-with-output_8_0.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:a72aac750b87ad7318f3caad198955aea63148e2bfa01db73fc7ccb9585304f8
-size 1151306
+oid sha256:1273f178881d3158052a8c494b40a5bb426a972130242bc0f84145c5bb2d98d6
+size 1150777
diff --git a/docs/notebooks/language-quantize-bert-with-output.rst b/docs/notebooks/language-quantize-bert-with-output.rst
index 555d4dc3a07..1635441d2ac 100644
--- a/docs/notebooks/language-quantize-bert-with-output.rst
+++ b/docs/notebooks/language-quantize-bert-with-output.rst
@@ -50,15 +50,7 @@ Table of contents:
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -98,14 +90,10 @@ Imports
.. parsed-literal::
- 2024-04-18 00:02:10.199327: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:02:10.234239: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:20:31.324929: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:20:31.359787: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:02:10.832965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-07 00:20:31.955321: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
@@ -207,12 +195,12 @@ PyTorch model formats are supported:
.. parsed-literal::
- WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.2.2+cpu. If you encounter issues, consider switching to torch==2.1.2
+ WARNING:nncf:NNCF provides best results with torch==2.2.*, while current torch version is 2.3.0+cpu. If you encounter issues, consider switching to torch==2.2.*
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
@@ -326,10 +314,6 @@ The optimization process contains the following steps:
.. parsed-literal::
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:50 ignored nodes were found by name in the NNCFGraph
@@ -492,16 +476,8 @@ Compare F1-score of FP32 and INT8 models
.. parsed-literal::
Checking the accuracy of the original model:
-
-
-.. parsed-literal::
-
F1 score: 0.9019
Checking the accuracy of the quantized model:
-
-
-.. parsed-literal::
-
F1 score: 0.8969
@@ -556,17 +532,9 @@ Frames Per Second (FPS) for images.
.. parsed-literal::
- PyTorch model on CPU: 0.075 seconds per sentence, SPS: 13.34
-
-
-.. parsed-literal::
-
- IR FP32 model in OpenVINO Runtime/AUTO: 0.020 seconds per sentence, SPS: 48.84
-
-
-.. parsed-literal::
-
- OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 113.21
+ PyTorch model on CPU: 0.072 seconds per sentence, SPS: 13.84
+ IR FP32 model in OpenVINO Runtime/AUTO: 0.021 seconds per sentence, SPS: 47.97
+ OpenVINO IR INT8 model in OpenVINO Runtime/AUTO: 0.009 seconds per sentence, SPS: 110.23
Finally, measure the inference performance of OpenVINO ``FP32`` and
@@ -586,7 +554,7 @@ in OpenVINO.
.. code:: ipython3
# Inference FP32 model (OpenVINO IR)
- !benchmark_app -m $ir_model_xml -shape [1,128],[1,128],[1,128] -d device.value -api sync
+ !benchmark_app -m $ir_model_xml -shape [1,128],[1,128],[1,128] -d {device.value} -api sync
.. parsed-literal::
@@ -594,32 +562,97 @@ in OpenVINO.
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
- [ WARNING ] Default duration 120 seconds is used for unknown device device.value
+ [ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
+ [ INFO ] AUTO
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
- [ ERROR ] Exception from src/inference/src/cpp/core.cpp:216:
- Exception from src/inference/src/dev/core_impl.cpp:556:
- Device with "device" name is not registered in the OpenVINO Runtime
-
- Traceback (most recent call last):
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
- supported_properties = benchmark.core.get_property(device, properties.supported_properties())
- RuntimeError: Exception from src/inference/src/cpp/core.cpp:216:
- Exception from src/inference/src/dev/core_impl.cpp:556:
- Device with "device" name is not registered in the OpenVINO Runtime
-
-
+ [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 19.23 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,?]
+ [ INFO ] attention_mask , 36 (node: attention_mask) : i64 / [...] / [1,?]
+ [ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,?]
+ [ INFO ] Model outputs:
+ [ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [ INFO ] Reshaping model: 'input_ids': [1,128], '36': [1,128], 'token_type_ids': [1,128]
+ [ INFO ] Reshape model took 5.65 ms
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,128]
+ [ INFO ] attention_mask , 36 (node: attention_mask) : i64 / [...] / [1,128]
+ [ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,128]
+ [ INFO ] Model outputs:
+ [ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 376.44 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] NETWORK_NAME: Model0
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] MULTI_DEVICE_PRIORITIES: CPU
+ [ INFO ] CPU:
+ [ INFO ] AFFINITY: Affinity.CORE
+ [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
+ [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
+ [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
+ [ INFO ] ENABLE_CPU_PINNING: True
+ [ INFO ] ENABLE_HYPER_THREADING: False
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
+ [ INFO ] INFERENCE_NUM_THREADS: 12
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] KV_CACHE_PRECISION:
+ [ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
+ [ INFO ] NETWORK_NAME: Model0
+ [ INFO ] NUM_STREAMS: 1
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] PERFORMANCE_HINT: LATENCY
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
+ [ INFO ] PERF_COUNT: NO
+ [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_PRIORITY: Priority.MEDIUM
+ [ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
+ [ WARNING ] No input files were given for input '36'!. This input will be filled with random values!
+ [ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
+ [ INFO ] Fill input 'input_ids' with random values
+ [ INFO ] Fill input '36' with random values
+ [ INFO ] Fill input 'token_type_ids' with random values
+ [Step 10/11] Measuring performance (Start inference synchronously, limits: 120000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 22.61 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:['CPU']
+ [ INFO ] Count: 6217 iterations
+ [ INFO ] Duration: 120004.55 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 19.20 ms
+ [ INFO ] Average: 19.21 ms
+ [ INFO ] Min: 18.57 ms
+ [ INFO ] Max: 23.34 ms
+ [ INFO ] Throughput: 51.81 FPS
.. code:: ipython3
# Inference INT8 model (OpenVINO IR)
- ! benchmark_app -m $compressed_model_xml -shape [1,128],[1,128],[1,128] -d device.value -api sync
+ ! benchmark_app -m $compressed_model_xml -shape [1,128],[1,128],[1,128] -d {device.value} -api sync
.. parsed-literal::
@@ -627,24 +660,89 @@ in OpenVINO.
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
- [ WARNING ] Default duration 120 seconds is used for unknown device device.value
+ [ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
+ [ INFO ] AUTO
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
- [ ERROR ] Exception from src/inference/src/cpp/core.cpp:216:
- Exception from src/inference/src/dev/core_impl.cpp:556:
- Device with "device" name is not registered in the OpenVINO Runtime
-
- Traceback (most recent call last):
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 166, in main
- supported_properties = benchmark.core.get_property(device, properties.supported_properties())
- RuntimeError: Exception from src/inference/src/cpp/core.cpp:216:
- Exception from src/inference/src/dev/core_impl.cpp:556:
- Device with "device" name is not registered in the OpenVINO Runtime
-
-
+ [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.LATENCY.
+ [Step 4/11] Reading model files
+ [ INFO ] Loading model files
+ [ INFO ] Read model took 24.76 ms
+ [ INFO ] Original model I/O parameters:
+ [ INFO ] Model inputs:
+ [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,?]
+ [ INFO ] 36 , attention_mask (node: attention_mask) : i64 / [...] / [1,?]
+ [ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,?]
+ [ INFO ] Model outputs:
+ [ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
+ [Step 5/11] Resizing model to match image sizes and given batch
+ [ INFO ] Model batch size: 1
+ [ INFO ] Reshaping model: 'input_ids': [1,128], '36': [1,128], 'token_type_ids': [1,128]
+ [ INFO ] Reshape model took 7.38 ms
+ [Step 6/11] Configuring input of the model
+ [ INFO ] Model inputs:
+ [ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,128]
+ [ INFO ] 36 , attention_mask (node: attention_mask) : i64 / [...] / [1,128]
+ [ INFO ] token_type_ids (node: token_type_ids) : i64 / [...] / [1,128]
+ [ INFO ] Model outputs:
+ [ INFO ] logits (node: __module.classifier/aten::linear/Add) : f32 / [...] / [1,2]
+ [Step 7/11] Loading the model to the device
+ [ INFO ] Compile model took 1183.71 ms
+ [Step 8/11] Querying optimal runtime parameters
+ [ INFO ] Model:
+ [ INFO ] NETWORK_NAME: Model0
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] PERFORMANCE_HINT: PerformanceMode.LATENCY
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] MULTI_DEVICE_PRIORITIES: CPU
+ [ INFO ] CPU:
+ [ INFO ] AFFINITY: Affinity.CORE
+ [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
+ [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
+ [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
+ [ INFO ] ENABLE_CPU_PINNING: True
+ [ INFO ] ENABLE_HYPER_THREADING: False
+ [ INFO ] EXECUTION_DEVICES: ['CPU']
+ [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
+ [ INFO ] INFERENCE_NUM_THREADS: 12
+ [ INFO ] INFERENCE_PRECISION_HINT:
+ [ INFO ] KV_CACHE_PRECISION:
+ [ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
+ [ INFO ] NETWORK_NAME: Model0
+ [ INFO ] NUM_STREAMS: 1
+ [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
+ [ INFO ] PERFORMANCE_HINT: LATENCY
+ [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
+ [ INFO ] PERF_COUNT: NO
+ [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_PRIORITY: Priority.MEDIUM
+ [ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
+ [Step 9/11] Creating infer requests and preparing input tensors
+ [ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
+ [ WARNING ] No input files were given for input '36'!. This input will be filled with random values!
+ [ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
+ [ INFO ] Fill input 'input_ids' with random values
+ [ INFO ] Fill input '36' with random values
+ [ INFO ] Fill input 'token_type_ids' with random values
+ [Step 10/11] Measuring performance (Start inference synchronously, limits: 120000 ms duration)
+ [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
+ [ INFO ] First inference took 15.91 ms
+ [Step 11/11] Dumping statistics report
+ [ INFO ] Execution Devices:['CPU']
+ [ INFO ] Count: 11978 iterations
+ [ INFO ] Duration: 120006.11 ms
+ [ INFO ] Latency:
+ [ INFO ] Median: 10.29 ms
+ [ INFO ] Average: 9.93 ms
+ [ INFO ] Min: 8.15 ms
+ [ INFO ] Max: 11.91 ms
+ [ INFO ] Throughput: 99.81 FPS
diff --git a/docs/notebooks/latency-tricks-with-output.rst b/docs/notebooks/latency-tricks-with-output.rst
deleted file mode 100644
index cc27188b158..00000000000
--- a/docs/notebooks/latency-tricks-with-output.rst
+++ /dev/null
@@ -1,869 +0,0 @@
-Performance tricks in OpenVINO for latency mode
-===============================================
-
-The goal of this notebook is to provide a step-by-step tutorial for
-improving performance for inferencing in a latency mode. Low latency is
-especially desired in real-time applications when the results are needed
-as soon as possible after the data appears. This notebook assumes
-computer vision workflow and uses
-`YOLOv5n `__ model. We will
-simulate a camera application that provides frames one by one.
-
-The performance tips applied in this notebook could be summarized in the
-following figure. Some of the steps below can be applied to any device
-at any stage, e.g., ``shared_memory``; some can be used only to specific
-devices, e.g., ``INFERENCE_NUM_THREADS`` to CPU. As the number of
-potential configurations is vast, we recommend looking at the steps
-below and then apply a trial-and-error approach. You can incorporate
-many hints simultaneously, like more inference threads + shared memory.
-It should give even better performance, but we recommend testing it
-anyway.
-
- **NOTE**: We especially recommend trying
- ``OpenVINO IR model + CPU + shared memory in latency mode`` or
- ``OpenVINO IR model + CPU + shared memory + more inference threads``.
-
-The quantization and pre-post-processing API are not included here as
-they change the precision (quantization) or processing graph
-(prepostprocessor). You can find examples of how to apply them to
-optimize performance on OpenVINO IR files in
-`optimize-preprocessing <../optimize-preprocessing>`__.
-
-|image0|
-
- **NOTE**: Many of the steps presented below will give you better
- performance. However, some of them may **not change anything** or
- even **worsen the performance** if they are strongly dependent on
- either the hardware or the model. Please run this notebook on your
- computer with your model to learn which of them makes sense in your
- case.
-
- All the following tricks were run with OpenVINO 2023.0. Future
- versions of OpenVINO may include various optimizations that may
- result in different performance.
-
-A similar notebook focused on the throughput mode is available
-`here `__.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Prerequisites <#prerequisites>`__
-- `Data <#data>`__
-- `Model <#model>`__
-- `Hardware <#hardware>`__
-- `Helper functions <#helper-functions>`__
-- `Optimizations <#optimizations>`__
-
- - `PyTorch model <#pytorch-model>`__
- - `ONNX model <#onnx-model>`__
- - `OpenVINO IR model <#openvino-ir-model>`__
- - `OpenVINO IR model on GPU <#openvino-ir-model-on-gpu>`__
- - `OpenVINO IR model + more inference
- threads <#openvino-ir-model--more-inference-threads>`__
- - `OpenVINO IR model in latency
- mode <#openvino-ir-model-in-latency-mode>`__
- - `OpenVINO IR model in latency mode + shared
- memory <#openvino-ir-model-in-latency-mode--shared-memory>`__
- - `Other tricks <#other-tricks>`__
-
-- `Performance comparison <#performance-comparison>`__
-- `Conclusions <#conclusions>`__
-
-.. |image0| image:: https://user-images.githubusercontent.com/4547501/229120774-01f4f972-424d-4280-8395-220dd432985a.png
-
-Prerequisites
--------------
-
-
-
-.. code:: ipython3
-
- import platform
-
- %pip install -q "openvino>=2023.1.0" seaborn "ultralytics<=8.0.178" onnx opencv-python --extra-index-url https://download.pytorch.org/whl/cpu
-
- if platform.system() != "Windows":
- %pip install -q "matplotlib>=3.4"
- else:
- %pip install -q "matplotlib>=3.4,<3.7"
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- import os
- import time
- from pathlib import Path
- from typing import Any, List, Tuple
-
- # Fetch `notebook_utils` module
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
-
- open("notebook_utils.py", "w").write(r.text)
- import notebook_utils as utils
-
-Data
-----
-
-
-
-We will use the same image of the dog sitting on a bicycle for all
-experiments below. The image is resized and preprocessed to fulfill the
-requirements of this particular object detection model.
-
-.. code:: ipython3
-
- import numpy as np
- import cv2
-
- IMAGE_WIDTH = 640
- IMAGE_HEIGHT = 480
-
- # load image
- image = utils.load_image("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg")
- image = cv2.resize(image, dsize=(IMAGE_WIDTH, IMAGE_HEIGHT), interpolation=cv2.INTER_AREA)
-
- # preprocess it for YOLOv5
- input_image = image / 255.0
- input_image = np.transpose(input_image, axes=(2, 0, 1))
- input_image = np.expand_dims(input_image, axis=0)
-
- # show the image
- utils.show_array(image)
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg
-
-
-
-
-.. parsed-literal::
-
-
-
-
-
-Model
------
-
-
-
-We decided to go with
-`YOLOv5n `__, one of the
-state-of-the-art object detection models, easily available through the
-PyTorch Hub and small enough to see the difference in performance.
-
-.. code:: ipython3
-
- import torch
- from IPython.utils import io
-
- # directory for all models
- base_model_dir = Path("model")
-
- model_name = "yolov5n"
- model_path = base_model_dir / model_name
-
- # load YOLOv5n from PyTorch Hub
- pytorch_model = torch.hub.load("ultralytics/yolov5", "custom", path=model_path, device="cpu", skip_validation=True)
- # don't print full model architecture
- with io.capture_output():
- pytorch_model.eval()
-
-
-.. parsed-literal::
-
- Using cache found in /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master
-
-
-.. parsed-literal::
-
- YOLOv5 🚀 2023-4-21 Python-3.8.10 torch-2.2.2+cpu CPU
-
-
-
-.. parsed-literal::
-
- requirements: /opt/home/k8sworker/.cache/torch/hub/requirements.txt not found, check failed.
-
-
-.. parsed-literal::
-
- Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5n.pt to model/yolov5n.pt...
-
-
-.. parsed-literal::
-
-
- 0%| | 0.00/3.87M [00:00 float:
- """
- Helper function for benchmarking the model. It measures the time and prints results.
- """
- # measure the first inference separately - it may be slower as it contains also initialization
- start = time.perf_counter()
- model(input_data)
- end = time.perf_counter()
- first_infer_time = end - start
- print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")
-
- # benchmarking
- start = time.perf_counter()
- for _ in range(INFER_NUMBER):
- model(input_data)
- end = time.perf_counter()
-
- # elapsed time
- infer_time = end - start
-
- # print second per image and FPS
- mean_infer_time = infer_time / INFER_NUMBER
- mean_fps = INFER_NUMBER / infer_time
- print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")
-
- return mean_infer_time
-
-The following functions aim to post-process results and draw boxes on
-the image.
-
-.. code:: ipython3
-
- # https://gist.github.com/AruniRC/7b3dadd004da04c80198557db5da4bda
- classes = [
- "person",
- "bicycle",
- "car",
- "motorcycle",
- "airplane",
- "bus",
- "train",
- "truck",
- "boat",
- "traffic light",
- "fire hydrant",
- "stop sign",
- "parking meter",
- "bench",
- "bird",
- "cat",
- "dog",
- "horse",
- "sheep",
- "cow",
- "elephant",
- "bear",
- "zebra",
- "giraffe",
- "backpack",
- "umbrella",
- "handbag",
- "tie",
- "suitcase",
- "frisbee",
- "skis",
- "snowboard",
- "sports ball",
- "kite",
- "baseball bat",
- "baseball glove",
- "skateboard",
- "surfboard",
- "tennis racket",
- "bottle",
- "wine glass",
- "cup",
- "fork",
- "knife",
- "spoon",
- "bowl",
- "banana",
- "apple",
- "sandwich",
- "orange",
- "broccoli",
- "carrot",
- "hot dog",
- "pizza",
- "donut",
- "cake",
- "chair",
- "couch",
- "potted plant",
- "bed",
- "dining table",
- "toilet",
- "tv",
- "laptop",
- "mouse",
- "remote",
- "keyboard",
- "cell phone",
- "microwave",
- "oven",
- "oaster",
- "sink",
- "refrigerator",
- "book",
- "clock",
- "vase",
- "scissors",
- "teddy bear",
- "hair drier",
- "toothbrush",
- ]
-
- # Colors for the classes above (Rainbow Color Map).
- colors = cv2.applyColorMap(
- src=np.arange(0, 255, 255 / len(classes), dtype=np.float32).astype(np.uint8),
- colormap=cv2.COLORMAP_RAINBOW,
- ).squeeze()
-
-
- def postprocess(detections: np.ndarray) -> List[Tuple]:
- """
- Postprocess the raw results from the model.
- """
- # candidates - probability > 0.25
- detections = detections[detections[..., 4] > 0.25]
-
- boxes = []
- labels = []
- scores = []
- for obj in detections:
- xmin, ymin, ww, hh = obj[:4]
- score = obj[4]
- label = np.argmax(obj[5:])
- # Create a box with pixels coordinates from the box with normalized coordinates [0,1].
- boxes.append(tuple(map(int, (xmin - ww // 2, ymin - hh // 2, ww, hh))))
- labels.append(int(label))
- scores.append(float(score))
-
- # Apply non-maximum suppression to get rid of many overlapping entities.
- # See https://paperswithcode.com/method/non-maximum-suppression
- # This algorithm returns indices of objects to keep.
- indices = cv2.dnn.NMSBoxes(bboxes=boxes, scores=scores, score_threshold=0.25, nms_threshold=0.5)
-
- # If there are no boxes.
- if len(indices) == 0:
- return []
-
- # Filter detected objects.
- return [(labels[idx], scores[idx], boxes[idx]) for idx in indices.flatten()]
-
-
- def draw_boxes(img: np.ndarray, boxes):
- """
- Draw detected boxes on the image.
- """
- for label, score, box in boxes:
- # Choose color for the label.
- color = tuple(map(int, colors[label]))
- # Draw a box.
- x2 = box[0] + box[2]
- y2 = box[1] + box[3]
- cv2.rectangle(img=img, pt1=box[:2], pt2=(x2, y2), color=color, thickness=2)
-
- # Draw a label name inside the box.
- cv2.putText(
- img=img,
- text=f"{classes[label]} {score:.2f}",
- org=(box[0] + 10, box[1] + 20),
- fontFace=cv2.FONT_HERSHEY_COMPLEX,
- fontScale=img.shape[1] / 1200,
- color=color,
- thickness=1,
- lineType=cv2.LINE_AA,
- )
-
-
- def show_result(results: np.ndarray):
- """
- Postprocess the raw results, draw boxes and show the image.
- """
- output_img = image.copy()
-
- detections = postprocess(results)
- draw_boxes(output_img, detections)
-
- utils.show_array(output_img)
-
-Optimizations
--------------
-
-
-
-Below, we present the performance tricks for faster inference in the
-latency mode. We release resources after every benchmarking to be sure
-the same amount of resource is available for every experiment.
-
-PyTorch model
-~~~~~~~~~~~~~
-
-
-
-First, we’re benchmarking the original PyTorch model without any
-optimizations applied. We will treat it as our baseline.
-
-.. code:: ipython3
-
- import torch
-
- with torch.no_grad():
- result = pytorch_model(torch.as_tensor(input_image)).detach().numpy()[0]
- show_result(result)
- pytorch_infer_time = benchmark_model(
- pytorch_model,
- input_data=torch.as_tensor(input_image).float(),
- benchmark_name="PyTorch model",
- )
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg
-
-
-.. parsed-literal::
-
- PyTorch model on CPU. First inference time: 0.0269 seconds
-
-
-.. parsed-literal::
-
- PyTorch model on CPU: 0.0213 seconds per image (47.06 FPS)
-
-
-ONNX model
-~~~~~~~~~~
-
-
-
-The first optimization is exporting the PyTorch model to ONNX and
-running it in OpenVINO. It’s possible, thanks to the ONNX frontend. It
-means we don’t necessarily have to convert the model to Intermediate
-Representation (IR) to leverage the OpenVINO Runtime.
-
-.. code:: ipython3
-
- onnx_path = base_model_dir / Path(f"{model_name}_{IMAGE_WIDTH}_{IMAGE_HEIGHT}").with_suffix(".onnx")
-
- # export PyTorch model to ONNX if it doesn't already exist
- if not onnx_path.exists():
- dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
- torch.onnx.export(pytorch_model, dummy_input, onnx_path)
-
- # load and compile in OpenVINO
- onnx_model = core.read_model(onnx_path)
- onnx_model = core.compile_model(onnx_model, device_name="CPU")
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:514: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
- /opt/home/k8sworker/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py:64: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:
-
-
-.. code:: ipython3
-
- result = onnx_model(input_image)[onnx_model.output(0)][0]
- show_result(result)
- onnx_infer_time = benchmark_model(model=onnx_model, input_data=input_image, benchmark_name="ONNX model")
-
- del onnx_model # release resources
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg
-
-
-.. parsed-literal::
-
- ONNX model on CPU. First inference time: 0.0173 seconds
-
-
-.. parsed-literal::
-
- ONNX model on CPU: 0.0136 seconds per image (73.79 FPS)
-
-
-OpenVINO IR model
-~~~~~~~~~~~~~~~~~
-
-
-
-Let’s convert the ONNX model to OpenVINO Intermediate Representation
-(IR) FP16 and run it. Reducing the precision is one of the well-known
-methods for faster inference provided the hardware that supports lower
-precision, such as FP16 or even INT8. If the hardware doesn’t support
-lower precision, the model will be inferred in FP32 automatically. We
-could also use quantization (INT8), but we should experience a little
-accuracy drop. That’s why we skip that step in this notebook.
-
-.. code:: ipython3
-
- ov_model = ov.convert_model(onnx_path)
- # save the model on disk
- ov.save_model(ov_model, str(onnx_path.with_suffix(".xml")))
-
- ov_cpu_model = core.compile_model(ov_model, device_name="CPU")
-
- result = ov_cpu_model(input_image)[ov_cpu_model.output(0)][0]
- show_result(result)
- ov_cpu_infer_time = benchmark_model(model=ov_cpu_model, input_data=input_image, benchmark_name="OpenVINO model")
-
- del ov_cpu_model # release resources
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg
-
-
-.. parsed-literal::
-
- OpenVINO model on CPU. First inference time: 0.0157 seconds
-
-
-.. parsed-literal::
-
- OpenVINO model on CPU: 0.0122 seconds per image (81.74 FPS)
-
-
-OpenVINO IR model on GPU
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Usually, a GPU device is faster than a CPU, so let’s run the above model
-on the GPU. Please note you need to have an Intel GPU and `install
-drivers `__
-to be able to run this step. In addition, offloading to the GPU helps
-reduce CPU load and memory consumption, allowing it to be left for
-routine processes. If you cannot observe a faster inference on GPU, it
-may be because the model is too light to benefit from massive parallel
-execution.
-
-.. code:: ipython3
-
- ov_gpu_infer_time = 0.0
- if "GPU" in core.available_devices:
- ov_gpu_model = core.compile_model(ov_model, device_name="GPU")
-
- result = ov_gpu_model(input_image)[ov_gpu_model.output(0)][0]
- show_result(result)
- ov_gpu_infer_time = benchmark_model(
- model=ov_gpu_model,
- input_data=input_image,
- benchmark_name="OpenVINO model",
- device_name="GPU",
- )
-
- del ov_gpu_model # release resources
-
-OpenVINO IR model + more inference threads
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-There is a possibility to add a config for any device (CPU in this
-case). We will increase the number of threads to an equal number of our
-cores. There are `more
-options `__
-to be changed, so it’s worth playing with them to see what works best in
-our case. In some cases, this optimization may worsen the performance.
-If it is the case, don’t use it.
-
-.. code:: ipython3
-
- num_cores = os.cpu_count()
-
- ov_cpu_config_model = core.compile_model(ov_model, device_name="CPU", config={"INFERENCE_NUM_THREADS": num_cores})
-
- result = ov_cpu_config_model(input_image)[ov_cpu_config_model.output(0)][0]
- show_result(result)
- ov_cpu_config_infer_time = benchmark_model(
- model=ov_cpu_config_model,
- input_data=input_image,
- benchmark_name="OpenVINO model + more threads",
- )
-
- del ov_cpu_config_model # release resources
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg
-
-
-.. parsed-literal::
-
- OpenVINO model + more threads on CPU. First inference time: 0.0157 seconds
-
-
-.. parsed-literal::
-
- OpenVINO model + more threads on CPU: 0.0123 seconds per image (81.20 FPS)
-
-
-OpenVINO IR model in latency mode
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-OpenVINO offers a virtual device called
-`AUTO `__,
-which can select the best device for us based on a performance hint.
-There are three different hints: ``LATENCY``, ``THROUGHPUT``, and
-``CUMULATIVE_THROUGHPUT``. As this notebook is focused on the latency
-mode, we will use ``LATENCY``. The above hints can be used with other
-devices as well.
-
-.. code:: ipython3
-
- ov_auto_model = core.compile_model(ov_model, device_name="AUTO", config={"PERFORMANCE_HINT": "LATENCY"})
-
- result = ov_auto_model(input_image)[ov_auto_model.output(0)][0]
- show_result(result)
- ov_auto_infer_time = benchmark_model(
- model=ov_auto_model,
- input_data=input_image,
- benchmark_name="OpenVINO model",
- device_name="AUTO",
- )
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg
-
-
-.. parsed-literal::
-
- OpenVINO model on AUTO. First inference time: 0.0157 seconds
-
-
-.. parsed-literal::
-
- OpenVINO model on AUTO: 0.0124 seconds per image (80.65 FPS)
-
-
-OpenVINO IR model in latency mode + shared memory
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-OpenVINO is a C++ toolkit with Python wrappers (API). The default
-behavior in the Python API is copying the input to the additional buffer
-and then running processing in C++, which prevents many
-multiprocessing-related issues. However, it also increases time cost. We
-can create a tensor with enabled shared memory (keeping in mind we
-cannot overwrite our input), save time for copying and improve the
-performance!
-
-.. code:: ipython3
-
- # it must be assigned to a variable, not to be garbage collected
- c_input_image = np.ascontiguousarray(input_image, dtype=np.float32)
- input_tensor = ov.Tensor(c_input_image, shared_memory=True)
-
- result = ov_auto_model(input_tensor)[ov_auto_model.output(0)][0]
- show_result(result)
- ov_auto_shared_infer_time = benchmark_model(
- model=ov_auto_model,
- input_data=input_tensor,
- benchmark_name="OpenVINO model + shared memory",
- device_name="AUTO",
- )
-
- del ov_auto_model # release resources
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg
-
-
-.. parsed-literal::
-
- OpenVINO model + shared memory on AUTO. First inference time: 0.0113 seconds
-
-
-.. parsed-literal::
-
- OpenVINO model + shared memory on AUTO: 0.0054 seconds per image (186.74 FPS)
-
-
-Other tricks
-~~~~~~~~~~~~
-
-
-
-There are other tricks for performance improvement, such as quantization
-and pre-post-processing or dedicated to throughput mode. To get even
-more from your model, please visit
-`optimize-preprocessing <../optimize-preprocessing>`__, and
-`throughput-tricks `__.
-
-Performance comparison
-----------------------
-
-
-
-The following graphical comparison is valid for the selected model and
-hardware simultaneously. If you cannot see any improvement between some
-steps, just skip them.
-
-.. code:: ipython3
-
- %matplotlib inline
-
-.. code:: ipython3
-
- from matplotlib import pyplot as plt
-
- labels = [
- "PyTorch model",
- "ONNX model",
- "OpenVINO IR model",
- "OpenVINO IR model on GPU",
- "OpenVINO IR model + more inference threads",
- "OpenVINO IR model in latency mode",
- "OpenVINO IR model in latency mode + shared memory",
- ]
- # make them milliseconds
- times = list(
- map(
- lambda x: 1000 * x,
- [
- pytorch_infer_time,
- onnx_infer_time,
- ov_cpu_infer_time,
- ov_gpu_infer_time,
- ov_cpu_config_infer_time,
- ov_auto_infer_time,
- ov_auto_shared_infer_time,
- ],
- )
- )
-
- bar_colors = colors[::10] / 255.0
-
- fig, ax = plt.subplots(figsize=(16, 8))
- ax.bar(labels, times, color=bar_colors)
-
- ax.set_ylabel("Inference time [ms]")
- ax.set_title("Performance difference")
-
- plt.xticks(rotation="vertical")
- plt.show()
-
-
-
-.. image:: latency-tricks-with-output_files/latency-tricks-with-output_31_0.png
-
-
-Conclusions
------------
-
-
-
-We already showed the steps needed to improve the performance of an
-object detection model. Even if you experience much better performance
-after running this notebook, please note this may not be valid for every
-hardware or every model. For the most accurate results, please use
-``benchmark_app`` `command-line
-tool `__.
-Note that ``benchmark_app`` cannot measure the impact of some tricks
-above, e.g., shared memory.
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_15_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_18_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_20_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_24_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_26_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg
deleted file mode 100644
index ab32fedd0f3..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_28_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:84e4f91c248768c2ea746240e307041396099f0d52fdb89b0179fa72e353894a
-size 162715
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_31_0.png b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_31_0.png
deleted file mode 100644
index c2b9c5013c2..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_31_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e50d8a22c26feebfe2a2d322d21b370b0610cbf00740d0a5808a2644d62f7028
-size 57013
diff --git a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg b/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg
deleted file mode 100644
index 510b092f676..00000000000
--- a/docs/notebooks/latency-tricks-with-output_files/latency-tricks-with-output_5_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:41c502fdff24ada81c63ccfca7d9153ea368b1eb3330caa03afa3281c35e4484
-size 155828
diff --git a/docs/notebooks/latent-consistency-models-image-generation-with-output.rst b/docs/notebooks/latent-consistency-models-image-generation-with-output.rst
index 5f07a0906ff..d4dd8932f57 100644
--- a/docs/notebooks/latent-consistency-models-image-generation-with-output.rst
+++ b/docs/notebooks/latent-consistency-models-image-generation-with-output.rst
@@ -553,14 +553,6 @@ decoded by the decoder part of the variational auto encoder.
return_tensors="pt",
)
text_input_ids = text_inputs.input_ids
- untruncated_ids = self.tokenizer(prompt, padding="longest", return_tensors="pt").input_ids
-
- if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(text_input_ids, untruncated_ids):
- removed_text = self.tokenizer.batch_decode(untruncated_ids[:, self.tokenizer.model_max_length - 1 : -1])
- logger.warning(
- "The following part of your input was truncated because CLIP can only handle sequences up to"
- f" {self.tokenizer.model_max_length} tokens: {removed_text}"
- )
prompt_embeds = self.text_encoder(text_input_ids, share_inputs=True, share_outputs=True)
prompt_embeds = torch.from_numpy(prompt_embeds[0])
diff --git a/docs/notebooks/latent-consistency-models-optimum-demo-with-output.rst b/docs/notebooks/latent-consistency-models-optimum-demo-with-output.rst
index 7f0e13ffd06..9b2eb557682 100644
--- a/docs/notebooks/latent-consistency-models-optimum-demo-with-output.rst
+++ b/docs/notebooks/latent-consistency-models-optimum-demo-with-output.rst
@@ -56,15 +56,7 @@ Install required packages
.. parsed-literal::
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -129,14 +121,10 @@ https://huggingface.co/docs/diffusers/en/api/pipelines/latent_consistency_models
.. parsed-literal::
- 2024-04-18 00:04:08.916564: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:04:08.953118: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:26:31.942146: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:26:31.978132: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:04:09.550802: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-07 00:26:32.469596: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@@ -149,7 +137,7 @@ https://huggingface.co/docs/diffusers/en/api/pipelines/latent_consistency_models
prompt = "A cute squirrel in the forest, portrait, 8k"
- image = pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0).images[0]
+ image = pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0, height=512, width=512).images[0]
image.save("image_standard_pipeline.png")
image
@@ -217,10 +205,15 @@ and there is no need to do it manually
.. code:: ipython3
from optimum.intel.openvino import OVLatentConsistencyModelPipeline
+ from pathlib import Path
- ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", export=True, compile=False)
- ov_pipeline.reshape(batch_size=1, height=768, width=768, num_images_per_prompt=1)
- ov_pipeline.save_pretrained("./openvino_ir")
+ if not Path("./openvino_ir").exists():
+ ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", height=512, width=512, export=True, compile=False)
+ ov_pipeline.save_pretrained("./openvino_ir")
+ else:
+ ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("./openvino_ir", export=False, compile=False)
+
+ ov_pipeline.reshape(batch_size=1, height=512, width=512, num_images_per_prompt=1)
.. parsed-literal::
@@ -231,11 +224,7 @@ and there is no need to do it manually
.. parsed-literal::
Framework not specified. Using pt to export the model.
-
-
-.. parsed-literal::
-
- Keyword arguments {'subfolder': '', 'trust_remote_code': False} are not expected by StableDiffusionPipeline and will be ignored.
+ Keyword arguments {'subfolder': '', 'token': None, 'trust_remote_code': False} are not expected by StableDiffusionPipeline and will be ignored.
@@ -246,7 +235,7 @@ and there is no need to do it manually
.. parsed-literal::
- Using framework PyTorch: 2.2.2+cpu
+ Using framework PyTorch: 2.3.0+cpu
.. parsed-literal::
@@ -257,22 +246,58 @@ and there is no need to do it manually
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
+ Using framework PyTorch: 2.3.0+cpu
+ Using framework PyTorch: 2.3.0+cpu
+ Using framework PyTorch: 2.3.0+cpu
+
+
.. parsed-literal::
- Using framework PyTorch: 2.2.2+cpu
+ OVLatentConsistencyModelPipeline {
+ "_class_name": "OVLatentConsistencyModelPipeline",
+ "_diffusers_version": "0.24.0",
+ "feature_extractor": [
+ "transformers",
+ "CLIPImageProcessor"
+ ],
+ "requires_safety_checker": true,
+ "safety_checker": [
+ "stable_diffusion",
+ "StableDiffusionSafetyChecker"
+ ],
+ "scheduler": [
+ "diffusers",
+ "LCMScheduler"
+ ],
+ "text_encoder": [
+ "optimum",
+ "OVModelTextEncoder"
+ ],
+ "text_encoder_2": [
+ null,
+ null
+ ],
+ "tokenizer": [
+ "transformers",
+ "CLIPTokenizer"
+ ],
+ "unet": [
+ "optimum",
+ "OVModelUnet"
+ ],
+ "vae_decoder": [
+ "optimum",
+ "OVModelVaeDecoder"
+ ],
+ "vae_encoder": [
+ "optimum",
+ "OVModelVaeEncoder"
+ ]
+ }
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
-
-
-.. parsed-literal::
-
- Using framework PyTorch: 2.2.2+cpu
-
.. code:: ipython3
@@ -283,20 +308,8 @@ and there is no need to do it manually
.. parsed-literal::
Compiling the vae_decoder to CPU ...
-
-
-.. parsed-literal::
-
Compiling the unet to CPU ...
-
-
-.. parsed-literal::
-
Compiling the vae_encoder to CPU ...
-
-
-.. parsed-literal::
-
Compiling the text_encoder to CPU ...
@@ -304,7 +317,7 @@ and there is no need to do it manually
prompt = "A cute squirrel in the forest, portrait, 8k"
- image_ov = ov_pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0).images[0]
+ image_ov = ov_pipeline(prompt=prompt, num_inference_steps=4, guidance_scale=8.0, height=512, width=512).images[0]
image_ov.save("image_opt.png")
image_ov
diff --git a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.jpg b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.jpg
index 46a6211f019..1b071ec50e5 100644
--- a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.jpg
+++ b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:978131b23cda8ee5f97efcd3c11e8853158abb75b93be87c596b70987c0abab4
-size 77604
+oid sha256:f55f400d3c6132e0208b817685c99756778f625b48a68f0a061909eb21b17b73
+size 30125
diff --git a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.png b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.png
index cb21646d5ce..d37b8ec4365 100644
--- a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.png
+++ b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_15_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:156d18fc303a7c9a92e356d1da39456c34ee8719291cc88cb225e6081c6c8a37
-size 1004233
+oid sha256:b73d5a12608ef2be6bed0c105a19e7375b187853f502d6659c3fd84e0d968759
+size 416282
diff --git a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.jpg b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.jpg
index 96fb046ba7e..a0e8857a3f6 100644
--- a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.jpg
+++ b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.jpg
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:b22f2298a9e2ddfd16462395c9d44657c7e8fd719eac2376f2dde5799e18a311
-size 74573
+oid sha256:66b5aca06c991e57d8cf17b73f6260878086185c9a7f5901d054fe53637adf3a
+size 36310
diff --git a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.png b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.png
index d801259215f..96fbe02457f 100644
--- a/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.png
+++ b/docs/notebooks/latent-consistency-models-optimum-demo-with-output_files/latent-consistency-models-optimum-demo-with-output_8_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:36c4842ee6b9a0232ddc779aa24e05bcca0dfd9bfa018a4e77c9c990e59b4383
-size 1004628
+oid sha256:bfb27d57b247ea9ea90b3a6d194f3ad1d874c657ad6ec204b0fffc0623c09442
+size 454841
diff --git a/docs/notebooks/legacy-mo-convert-to-openvino-with-output.rst b/docs/notebooks/legacy-mo-convert-to-openvino-with-output.rst
index eaf936419c5..a5a927f2851 100644
--- a/docs/notebooks/legacy-mo-convert-to-openvino-with-output.rst
+++ b/docs/notebooks/legacy-mo-convert-to-openvino-with-output.rst
@@ -119,7 +119,7 @@ documentation.
conversion into IR. The legacy Frontend is Python
based and is available for TensorFlow*, ONNX*, MXNet*,
Caffe*, and Kaldi* models.
- --input_model INPUT_MODEL, -w INPUT_MODEL, -m INPUT_MODEL
+ --input_model INPUT_MODEL, -m INPUT_MODEL, -w INPUT_MODEL
Tensorflow*: a file with a pre-trained model (binary
or text .pb file after freezing). Caffe*: a model
proto file with model weights.
@@ -735,19 +735,13 @@ NLP model from Hugging Face and export it in ONNX format:
.. parsed-literal::
- 2024-04-17 23:28:06.943236: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-17 23:28:06.978999: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-06 23:47:34.384632: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-06 23:47:34.420652: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-17 23:28:07.628516: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:246: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
+ 2024-05-06 23:47:35.064940: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
mask, torch.tensor(torch.finfo(scores.dtype).min)
@@ -1009,13 +1003,9 @@ To convert a model to OpenVINO IR, use the following command:
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
-
-
-.. parsed-literal::
-
[ SUCCESS ] Generated IR version 11 model.
- [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
- [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
+ [ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.xml
+ [ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/convert-to-openvino/model/distilbert.bin
.. code:: ipython3
@@ -1109,13 +1099,9 @@ guide `__.
-You can get additional inference speed improvement with `Dynamic
-Quantization of activations and KV-cache
-quantization `__.
+.. code:: ipython3
+
+ from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
+
+ ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
+
+ ov_llm = HuggingFacePipeline.from_model_id(
+ model_id=model_path,
+ task="text-generation",
+ backend="openvino",
+ model_kwargs={"device": device.value, "ov_config": ov_config},
+ pipeline_kwargs={"max_new_tokens": 1024},
+ )
+
+
+.. parsed-literal::
+
+ 2024-05-01 12:57:42.013703: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-01 12:57:42.015389: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-05-01 12:57:42.049792: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-05-01 12:57:42.050591: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
+ 2024-05-01 12:57:42.819557: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
+ warn("The installed version of bitsandbytes was compiled without GPU support. "
+
+
+.. parsed-literal::
+
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
+ INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
+
+
+.. parsed-literal::
+
+ No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
+ WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
+ PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
+ Python 3.8.18 (you have 3.8.10)
+ Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
+ Memory-efficient attention, SwiGLU, sparse and more won't be available.
+ Set XFORMERS_MORE_DETAILS=1 for more details
+ Compiling the model to CPU ...
+
+
+You can get additional inference speed improvement with [Dynamic
+Quantization of activations and KV-cache quantization] on
+CPU(https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-hf.html#enabling-openvino-runtime-optimizations).
These options can be enabled with ``ov_config`` as follows:
.. code:: ipython3
@@ -272,18 +317,6 @@ These options can be enabled with ``ov_config`` as follows:
"CACHE_DIR": "",
}
-.. code:: ipython3
-
- from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
-
- ov_llm = HuggingFacePipeline.from_model_id(
- model_id=model_path,
- task="text-generation",
- backend="openvino",
- model_kwargs={"device": device.value, "ov_config": ov_config},
- pipeline_kwargs={"max_new_tokens": 1024},
- )
-
Create agent
------------
@@ -321,11 +354,6 @@ prompt template.
agent_executor.invoke({"input": "Take 3 to the fifth power and multiply that by the sum of twelve and three"})
-.. parsed-literal::
-
- Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
-
-
.. parsed-literal::
diff --git a/docs/notebooks/llm-chatbot-with-output.rst b/docs/notebooks/llm-chatbot-with-output.rst
index 931bfeb371b..355ca8b5eee 100644
--- a/docs/notebooks/llm-chatbot-with-output.rst
+++ b/docs/notebooks/llm-chatbot-with-output.rst
@@ -78,18 +78,18 @@ Install required dependencies
.. code:: ipython3
- import shutil
+ import os
from pathlib import Path
import requests
-
+
# fetch model configuration
-
+
config_shared_path = Path("../../utils/llm_config.py")
config_dst_path = Path("llm_config.py")
-
+
if not config_dst_path.exists():
if config_shared_path.exists():
- shutil.copy(config_shared_path, config_dst_path)
+ os.symlink(config_shared_path, config_dst_path)
else:
r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/llm_config.py")
with open("llm_config.py", "w") as f:
@@ -146,7 +146,7 @@ The available options are:
.. code:: python
- ## login to huggingfacehub to get access to pretrained model
+ ## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@@ -156,6 +156,14 @@ The available options are:
except OSError:
notebook_login()
+- **phi3-mini-instruct<|end|>** - The Phi-3-Mini is a 3.8B parameters,
+ lightweight, state-of-the-art open model trained with the Phi-3
+ datasets that includes both synthetic data and the filtered publicly
+ available websites data with a focus on high-quality and reasoning
+ dense properties. More details about model can be found in `model
+ card `__,
+ `Microsoft blog `__ and `technical
+ report `__.
- **red-pajama-3b-chat** - A 2.8B parameter pre-trained language model
based on GPT-NEOX architecture. It was developed by Together Computer
and leaders from the open-source AI community. The model is
@@ -184,7 +192,7 @@ The available options are:
.. code:: python
- ## login to huggingfacehub to get access to pretrained model
+ ## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@@ -217,7 +225,41 @@ The available options are:
.. code:: python
- ## login to huggingfacehub to get access to pretrained model
+ ## login to huggingfacehub to get access to pretrained model
+
+ from huggingface_hub import notebook_login, whoami
+
+ try:
+ whoami()
+ print('Authorization token already provided')
+ except OSError:
+ notebook_login()
+
+- **llama-3-8b-instruct** - Llama 3 is an auto-regressive language
+ model that uses an optimized transformer architecture. The tuned
+ versions use supervised fine-tuning (SFT) and reinforcement learning
+ with human feedback (RLHF) to align with human preferences for
+ helpfulness and safety. The Llama 3 instruction tuned models are
+ optimized for dialogue use cases and outperform many of the available
+ open source chat models on common industry benchmarks. More details
+ about model can be found in `Meta blog
+ post `__, `model
+ website `__ and `model
+ card `__.
+ >\ **Note**: run model with demo, you will need to accept license
+ agreement. >You must be a registered user in Hugging Face Hub.
+ Please visit `HuggingFace model
+ card `__,
+ carefully read terms of usage and click accept button. You will need
+ to use an access token for the code below to run. For more
+ information on access tokens, refer to `this section of the
+ documentation `__.
+ >You can login on Hugging Face Hub in notebook environment, using
+ following code:
+
+.. code:: python
+
+ ## login to huggingfacehub to get access to pretrained model
from huggingface_hub import notebook_login, whoami
@@ -336,14 +378,14 @@ The available options are:
.. code:: ipython3
model_languages = list(SUPPORTED_LLM_MODELS)
-
+
model_language = widgets.Dropdown(
options=model_languages,
value=model_languages[0],
description="Model Language:",
disabled=False,
)
-
+
model_language
@@ -358,14 +400,14 @@ The available options are:
.. code:: ipython3
model_ids = list(SUPPORTED_LLM_MODELS[model_language.value])
-
+
model_id = widgets.Dropdown(
options=model_ids,
- value=model_ids[0],
+ value=model_ids[2],
description="Model:",
disabled=False,
)
-
+
model_id
@@ -373,7 +415,7 @@ The available options are:
.. parsed-literal::
- Dropdown(description='Model:', options=('tiny-llama-1b-chat', 'gemma-2b-it', 'red-pajama-3b-chat', 'gemma-7b-i…
+ Dropdown(description='Model:', index=2, options=('tiny-llama-1b-chat', 'gemma-2b-it', 'phi-3-mini-instruct', '…
@@ -385,7 +427,7 @@ The available options are:
.. parsed-literal::
- Selected model tiny-llama-1b-chat
+ Selected model phi-3-mini-instruct
Convert model using Optimum-CLI tool
@@ -418,8 +460,7 @@ that exported model should solve. For LLMs it will be
``text-generation-with-past``. If model initialization requires to use
remote code, ``--trust-remote-code`` flag additionally should be passed.
-Compress model weights
-----------------------
+<|end|>## Compress model weights
The `Weights
Compression `__
@@ -462,7 +503,7 @@ sacrifice of the model size and inference latency.
.. code:: ipython3
from IPython.display import Markdown, display
-
+
prepare_int4_model = widgets.Checkbox(
value=True,
description="Prepare INT4 model",
@@ -478,7 +519,7 @@ sacrifice of the model size and inference latency.
description="Prepare FP16 model",
disabled=False,
)
-
+
display(prepare_int4_model)
display(prepare_int8_model)
display(prepare_fp16_model)
@@ -507,14 +548,14 @@ We can now save floating point and compressed model variants
.. code:: ipython3
from pathlib import Path
-
+
pt_model_id = model_configuration["model_id"]
pt_model_name = model_id.value.split("-")[0]
fp16_model_dir = Path(model_id.value) / "FP16"
int8_model_dir = Path(model_id.value) / "INT8_compressed_weights"
int4_model_dir = Path(model_id.value) / "INT4_compressed_weights"
-
-
+
+
def convert_to_fp16():
if (fp16_model_dir / "openvino_model.xml").exists():
return
@@ -526,8 +567,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
-
-
+
+
def convert_to_int8():
if (int8_model_dir / "openvino_model.xml").exists():
return
@@ -540,8 +581,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
-
-
+
+
def convert_to_int4():
compression_configs = {
"zephyr-7b-beta": {
@@ -579,6 +620,11 @@ We can now save floating point and compressed model variants
"group_size": 128,
"ratio": 0.8,
},
+ "llama-3-8b-instruct": {
+ "sym": True,
+ "group_size": 128,
+ "ratio": 0.8,
+ },
"gemma-7b-it": {
"sym": True,
"group_size": 128,
@@ -601,7 +647,7 @@ We can now save floating point and compressed model variants
"ratio": 0.8,
},
}
-
+
model_compression_params = compression_configs.get(model_id.value, compression_configs["default"])
if (int4_model_dir / "openvino_model.xml").exists():
return
@@ -617,8 +663,8 @@ We can now save floating point and compressed model variants
display(Markdown("**Export command:**"))
display(Markdown(f"`{export_command}`"))
! $export_command
-
-
+
+
if prepare_fp16_model.value:
convert_to_fp16()
if prepare_int8_model.value:
@@ -626,58 +672,6 @@ We can now save floating point and compressed model variants
if prepare_int4_model.value:
convert_to_int4()
-
-
-**Export command:**
-
-
-
-``optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --task text-generation-with-past --weight-format int4 --group-size 128 --ratio 0.8 tiny-llama-1b-chat/INT4_compressed_weights``
-
-
-.. parsed-literal::
-
- 2024-04-11 11:48:29.180963: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-11 11:48:29.182830: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
- 2024-04-11 11:48:29.219152: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
- 2024-04-11 11:48:29.219549: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
- To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
- 2024-04-11 11:48:29.930190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
- warn("The installed version of bitsandbytes was compiled without GPU support. "
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
- WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
- PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
- Python 3.8.18 (you have 3.8.10)
- Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
- Memory-efficient attention, SwiGLU, sparse and more won't be available.
- Set XFORMERS_MORE_DETAILS=1 for more details
- INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
- No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
- Framework not specified. Using pt to export the model.
- Using the export variant default. Available variants are:
- - default: The default ONNX variant.
- Using framework PyTorch: 2.1.2+cpu
- Overriding 1 configuration item(s)
- - use_cache -> True
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
- warnings.warn(
- The cos_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
- The sin_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
- /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/optimum/exporters/openvino/model_patcher.py:311: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if sequence_length != 1:
- [2KMixed-Precision assignment ━━━━━━━━━━━━━━━━━━━━ 100% 154/154 • 0:00:11 • 0:00:00;0;104;181m0:00:01181m0:00:01
- INFO:nncf:Statistics of the bitwidth distribution:
- ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
- │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │
- ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
- │ 8 │ 30% (42 / 156) │ 20% (40 / 154) │
- ├────────────────┼─────────────────────────────┼────────────────────────────────────────┤
- │ 4 │ 70% (114 / 156) │ 80% (114 / 154) │
- ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
- [2KApplying Weight Compression ━━━━━━━━━━━━━━━━━━━ 100% 156/156 • 0:00:26 • 0:00:00;0;104;181m0:00:01181m0:00:02
-
-
Let’s compare model size for different compression types
.. code:: ipython3
@@ -685,7 +679,7 @@ Let’s compare model size for different compression types
fp16_weights = fp16_model_dir / "openvino_model.bin"
int8_weights = int8_model_dir / "openvino_model.bin"
int4_weights = int4_model_dir / "openvino_model.bin"
-
+
if fp16_weights.exists():
print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
@@ -697,7 +691,7 @@ Let’s compare model size for different compression types
.. parsed-literal::
- Size of model with INT4 compressed weights is 696.19 MB
+ Size of model with INT4 compressed weights is 2339.74 MB
Select device for inference and model variant
@@ -711,16 +705,16 @@ Select device for inference and model variant
.. code:: ipython3
import openvino as ov
-
+
core = ov.Core()
-
+
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="CPU",
description="Device:",
disabled=False,
)
-
+
device
@@ -744,14 +738,14 @@ variant of model weights and inference device
available_models.append("INT8")
if fp16_model_dir.exists():
available_models.append("FP16")
-
+
model_to_run = widgets.Dropdown(
options=available_models,
value=available_models[0],
description="Model to run:",
disabled=False,
)
-
+
model_to_run
@@ -803,7 +797,7 @@ guide `__
from transformers import AutoConfig, AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
-
+
if model_to_run.value == "INT4":
model_dir = int4_model_dir
elif model_to_run.value == "INT8":
@@ -811,17 +805,17 @@ guide `__
else:
model_dir = fp16_model_dir
print(f"Loading model from {model_dir}")
-
+
ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
-
+
# On a GPU device a model is executed in FP16 precision. For red-pajama-3b-chat model there known accuracy
# issues caused by this, which we avoid by setting precision hint to "f32".
if model_id.value == "red-pajama-3b-chat" and "GPU" in core.available_devices and device.value in ["GPU", "AUTO"]:
ov_config["INFERENCE_PRECISION_HINT"] = "f32"
-
+
model_name = model_configuration["model_id"]
tok = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
-
+
ov_model = OVModelForCausalLM.from_pretrained(
model_dir,
device=device.value,
@@ -830,6 +824,52 @@ guide `__
trust_remote_code=True,
)
+
+.. parsed-literal::
+
+ INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
+
+
+.. parsed-literal::
+
+ No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
+ 2024-04-23 22:13:04.208987: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-04-23 22:13:04.210866: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-04-23 22:13:04.245998: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-04-23 22:13:04.246894: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
+ 2024-04-23 22:13:04.941663: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
+ warn("The installed version of bitsandbytes was compiled without GPU support. "
+
+
+.. parsed-literal::
+
+ /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
+
+
+.. parsed-literal::
+
+ WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
+ PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)
+ Python 3.8.18 (you have 3.8.10)
+ Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
+ Memory-efficient attention, SwiGLU, sparse and more won't be available.
+ Set XFORMERS_MORE_DETAILS=1 for more details
+ Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
+
+
+.. parsed-literal::
+
+ Loading model from phi-3-mini-instruct/INT4_compressed_weights
+
+
+.. parsed-literal::
+
+ The argument `trust_remote_code` is to be used along with export=True. It will be ignored.
+ Compiling the model to CPU ...
+
+
.. code:: ipython3
tokenizer_kwargs = model_configuration.get("tokenizer_kwargs", {})
@@ -839,11 +879,6 @@ guide `__
print(tok.batch_decode(answer, skip_special_tokens=True)[0])
-.. parsed-literal::
-
- Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
-
-
.. parsed-literal::
2 + 2 = 4
@@ -889,14 +924,14 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
::
- playing: 0.5
- sleeping: 0.25
- eating: 0.15
- driving: 0.05
- flying: 0.05
+ playing: 0.5
+ sleeping: 0.25
+ eating: 0.15
+ driving: 0.05
+ flying: 0.05
- - **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as "playing."
- - **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as "playing," "sleeping," or "eating."
+ - **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as "playing."
+ - **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as "playing," "sleeping," or "eating."
- **High temperature** (e.g., 2.0): The AI model becomes more adventurous, increasing the chances of selecting less likely tokens, such as "driving" and "flying."
- ``Top-p``, also known as nucleus sampling, is a parameter used to
@@ -943,15 +978,15 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
StoppingCriteriaList,
TextIteratorStreamer,
)
-
-
+
+
model_name = model_configuration["model_id"]
start_message = model_configuration["start_message"]
history_template = model_configuration.get("history_template")
current_message_template = model_configuration.get("current_message_template")
stop_tokens = model_configuration.get("stop_tokens")
tokenizer_kwargs = model_configuration.get("tokenizer_kwargs", {})
-
+
chinese_examples = [
["你好!"],
["你是谁?"],
@@ -961,7 +996,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["给我讲一个年轻人奋斗创业最终取得成功的故事。"],
["给这个故事起一个标题。"],
]
-
+
english_examples = [
["Hello there! How are you doing?"],
["What is OpenVINO?"],
@@ -971,7 +1006,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["What are some common mistakes to avoid when writing code?"],
["Write a 100-word blog post on “Benefits of Artificial Intelligence and OpenVINO“"],
]
-
+
japanese_examples = [
["こんにちは！調子はどうですか?"],
["OpenVINOとは何ですか?"],
@@ -981,48 +1016,48 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
["コードを書くときに避けるべきよくある間違いは何ですか?"],
["人工知能と「OpenVINOの利点」について100語程度のブログ記事を書いてください。"],
]
-
+
examples = chinese_examples if (model_language.value == "Chinese") else japanese_examples if (model_language.value == "Japanese") else english_examples
-
+
max_new_tokens = 256
-
-
+
+
class StopOnTokens(StoppingCriteria):
def __init__(self, token_ids):
self.token_ids = token_ids
-
+
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
for stop_id in self.token_ids:
if input_ids[0][-1] == stop_id:
return True
return False
-
-
+
+
if stop_tokens is not None:
if isinstance(stop_tokens[0], str):
stop_tokens = tok.convert_tokens_to_ids(stop_tokens)
-
+
stop_tokens = [StopOnTokens(stop_tokens)]
-
-
+
+
def default_partial_text_processor(partial_text: str, new_text: str):
"""
helper for updating partially generated answer, used by default
-
+
Params:
partial_text: text buffer for storing previosly generated text
new_text: text update for the current step
Returns:
updated text string
-
+
"""
partial_text += new_text
return partial_text
-
-
+
+
text_processor = model_configuration.get("partial_text_processor", default_partial_text_processor)
-
-
+
+
def convert_history_to_token(history: List[Tuple[str, str]]):
"""
function for conversion history stored as list pairs of user and assistant messages to tokens according to model expected conversation template
@@ -1056,7 +1091,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
messages.append({"role": "user", "content": user_msg})
if model_msg:
messages.append({"role": "assistant", "content": model_msg})
-
+
input_token = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt")
else:
text = start_message + "".join(
@@ -1077,12 +1112,12 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
)
input_token = tok(text, return_tensors="pt", **tokenizer_kwargs).input_ids
return input_token
-
-
+
+
def user(message, history):
"""
callback function for updating user messages in interface on submit button click
-
+
Params:
message: current message
history: conversation history
@@ -1091,12 +1126,12 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
"""
# Append the user's message to the conversation history
return "", history + [[message, ""]]
-
-
+
+
def bot(history, temperature, top_p, top_k, repetition_penalty, conversation_id):
"""
callback function for running chatbot on submit button click
-
+
Params:
history: conversation history
temperature: parameter for control the level of creativity in AI-generated text.
@@ -1105,9 +1140,9 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
top_k: parameter for control the range of tokens considered by the AI model based on their cumulative probability, selecting number of tokens with highest probability.
repetition_penalty: parameter for penalizing tokens based on how frequently they occur in the text.
conversation_id: unique conversation identifier.
-
+
"""
-
+
# Construct the input message string for the model by concatenating the current system message and conversation history
# Tokenize the messages string
input_ids = convert_history_to_token(history)
@@ -1127,9 +1162,9 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
)
if stop_tokens is not None:
generate_kwargs["stopping_criteria"] = StoppingCriteriaList(stop_tokens)
-
+
stream_complete = Event()
-
+
def generate_and_signal_complete():
"""
genration function for single thread
@@ -1137,29 +1172,29 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
global start_time
ov_model.generate(**generate_kwargs)
stream_complete.set()
-
+
t1 = Thread(target=generate_and_signal_complete)
t1.start()
-
+
# Initialize an empty string to store the generated text
partial_text = ""
for new_text in streamer:
partial_text = text_processor(partial_text, new_text)
history[-1][1] = partial_text
yield history
-
-
+
+
def request_cancel():
ov_model.request.cancel()
-
-
+
+
def get_uuid():
"""
universal unique identifier for thread
"""
return str(uuid4())
-
-
+
+
with gr.Blocks(
theme=gr.themes.Soft(),
css=".disclaimer {font-variant-caps: all-small-caps;}",
@@ -1231,7 +1266,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
info="Penalize repetition — 1.0 to disable.",
)
gr.Examples(examples, inputs=msg, label="Click on any example and press the 'Submit' button")
-
+
submit_event = msg.submit(
fn=user,
inputs=[msg, chatbot],
@@ -1276,7 +1311,7 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
queue=False,
)
clear.click(lambda: None, None, chatbot, queue=False)
-
+
# if you are launching remotely, specify server_name and server_port
# demo.launch(server_name='your server name', server_port='server port in int')
# if you have any issue to launch on your platform, you can pass share=True to launch method:
@@ -1284,6 +1319,26 @@ answers.https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html
# it creates a publicly shareable link for the interface. Read more in the docs: https://gradio.app/docs/
demo.launch()
+
+.. parsed-literal::
+
+ Running on local URL: http://127.0.0.1:7860
+
+ To create a public link, set `share=True` in `launch()`.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
.. code:: ipython3
# please uncomment and run this cell for stopping gradio interface
diff --git a/docs/notebooks/llm-question-answering-with-output.rst b/docs/notebooks/llm-question-answering-with-output.rst
index 272d5c9b7a2..cb5ac180411 100644
--- a/docs/notebooks/llm-question-answering-with-output.rst
+++ b/docs/notebooks/llm-question-answering-with-output.rst
@@ -128,9 +128,54 @@ The available options are:
card `__,
`paper `__ and `release blog
post `__.
+- **llama-3-8b-instruct** - Llama 3 is an auto-regressive language
+ model that uses an optimized transformer architecture. The tuned
+ versions use supervised fine-tuning (SFT) and reinforcement learning
+ with human feedback (RLHF) to align with human preferences for
+ helpfulness and safety. The Llama 3 instruction tuned models are
+ optimized for dialogue use cases and outperform many of the available
+ open source chat models on common industry benchmarks. More details
+ about model can be found in `Meta blog
+ post `__, `model
+ website `__ and `model
+ card `__.
+ >\ **Note**: run model with demo, you will need to accept license
+ agreement. >You must be a registered user in Hugging Face Hub.
+ Please visit `HuggingFace model
+ card `__,
+ carefully read terms of usage and click accept button. You will need
+ to use an access token for the code below to run. For more
+ information on access tokens, refer to `this section of the
+ documentation `__.
+ >You can login on Hugging Face Hub in notebook environment, using
+ following code:
+
+.. code:: python
+
+ ## login to huggingfacehub to get access to pretrained model
+
+ from huggingface_hub import notebook_login, whoami
+
+ try:
+ whoami()
+ print('Authorization token already provided')
+ except OSError:
+ notebook_login()
.. code:: ipython3
+ from pathlib import Path
+ import requests
+
+ # Fetch `notebook_utils` module
+ r = requests.get(
+ url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
+ )
+ open("notebook_utils.py", "w").write(r.text)
+ from notebook_utils import download_file
+
+ if not Path("./config.py").exists():
+ download_file(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-question-answering/config.py")
from config import SUPPORTED_LLM_MODELS
import ipywidgets as widgets
@@ -164,7 +209,7 @@ The available options are:
.. parsed-literal::
- Selected model phi-2
+ Selected model llama-3-8b-instruct
Instantiate Model using Optimum Intel
@@ -307,10 +352,8 @@ compression.
import openvino as ov
import nncf
from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig
- from optimum.utils import NormalizedTextConfig, NormalizedConfigManager
import gc
- NormalizedConfigManager._conf["phi"] = NormalizedTextConfig
nncf.set_log_level(logging.ERROR)
@@ -354,6 +397,7 @@ compression.
"ratio": 0.5,
},
"dolly-v2-3b": {"sym": False, "group_size": 32, "ratio": 0.5},
+ "llama-3-8b-instruct": {"sym": True, "group_size": 128, "ratio": 1.0},
"default": {
"sym": False,
"group_size": 128,
@@ -385,14 +429,71 @@ compression.
.. parsed-literal::
- INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino
+ INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
.. parsed-literal::
- /home/ea/work/genai_env/lib/python3.8/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
- return torch._C._cuda_getDeviceCount() > 0
- No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
+ 2024-04-19 10:35:50.012050: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-04-19 10:35:50.025002: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-04-19 10:35:50.060073: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
+ 2024-04-19 10:35:50.060108: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
+ 2024-04-19 10:35:50.060134: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
+ 2024-04-19 10:35:50.068691: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
+ 2024-04-19 10:35:50.069448: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
+ 2024-04-19 10:35:51.045741: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
+ Framework not specified. Using pt to export the model.
+
+
+
+.. parsed-literal::
+
+ Loading checkpoint shards: 0%| | 0/4 [00:00 True
+ /home/ea/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ warnings.warn(
+ The cos_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
+ The sin_cached attribute will be removed in 4.39. Bear in mind that its contents changed in v4.38. Use the forward method of RoPE from now on instead. It is not used in the `LlamaAttention` class
+ /home/ea/miniconda3/lib/python3.11/site-packages/optimum/exporters/openvino/model_patcher.py:311: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if sequence_length != 1:
+
+
+
+.. parsed-literal::
+
+ Output()
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. raw:: html
+
+
+
+
+
+
+
+.. parsed-literal::
+
+ 19445
+
+
+
+Text Encoder
+~~~~~~~~~~~~
+
+
+
+The text-encoder is responsible for transforming the input prompt, for
+example, “a photo of an astronaut riding a horse” into an embedding
+space that can be understood by the U-Net. It is usually a simple
+transformer-based encoder that maps a sequence of input tokens to a
+sequence of latent text embeddings.
+
+.. code:: ipython3
+
+ text_encoder = pipe.text_encoder
+ text_encoder.eval()
+ text_encoder_2 = pipe.text_encoder_2
+ text_encoder_2.eval()
+
+ text_encoder.config.output_hidden_states = True
+ text_encoder.config.return_dict = False
+ text_encoder_2.config.output_hidden_states = True
+ text_encoder_2.config.return_dict = False
+
+ inputs = {"input_ids": torch.ones((1, 77), dtype=torch.long)}
+
+ input_info = prepare_input_info(inputs)
+
+ convert(text_encoder, TEXT_ENCODER_OV_PATH, inputs, input_info)
+ convert(text_encoder_2, TEXT_ENCODER_2_OV_PATH, inputs, input_info)
+
+ del text_encoder
+ del text_encoder_2
+ gc.collect()
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:86: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if input_shape[-1] > 1 or self.sliding_window is not None:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:162: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if past_key_values_length > 0:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
+
+
+.. parsed-literal::
+
+ INFO:nncf:Statistics of the bitwidth distribution:
+ ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
+ │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │
+ ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
+ │ 8 │ 100% (74 / 74) │ 100% (74 / 74) │
+ ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
+
+
+
+.. parsed-literal::
+
+ Output()
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. raw:: html
+
+
+
+
+
+
+
+.. parsed-literal::
+
+ 38600
+
+
+
+U-Net
+~~~~~
+
+
+
+The process of U-Net model conversion remains the same, like for
+original Stable Diffusion XL model.
+
+.. code:: ipython3
+
+ unet = pipe.unet
+ unet.eval()
+
+
+ class UnetWrapper(torch.nn.Module):
+ def __init__(self, unet):
+ super().__init__()
+ self.unet = unet
+
+ def forward(
+ self,
+ sample=None,
+ timestep=None,
+ encoder_hidden_states=None,
+ text_embeds=None,
+ time_ids=None,
+ ):
+ return self.unet.forward(
+ sample,
+ timestep,
+ encoder_hidden_states,
+ added_cond_kwargs={"text_embeds": text_embeds, "time_ids": time_ids},
+ )
+
+
+ inputs = {
+ "sample": torch.rand([2, 4, 128, 128], dtype=torch.float32),
+ "timestep": torch.from_numpy(np.array(1, dtype=float)),
+ "encoder_hidden_states": torch.rand([2, 77, 2048], dtype=torch.float32),
+ "text_embeds": torch.rand([2, 1280], dtype=torch.float32),
+ "time_ids": torch.rand([2, 6], dtype=torch.float32),
+ }
+
+ input_info = prepare_input_info(inputs)
+
+ w_unet = UnetWrapper(unet)
+ convert(w_unet, UNET_OV_PATH, inputs, input_info)
+
+ del w_unet, unet
+ gc.collect()
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/unets/unet_2d_condition.py:1110: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if dim % default_overall_up_factor != 0:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/downsampling.py:137: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/downsampling.py:146: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if hidden_states.shape[0] >= 64:
+
+
+.. parsed-literal::
+
+ INFO:nncf:Statistics of the bitwidth distribution:
+ ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
+ │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │
+ ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
+ │ 8 │ 100% (794 / 794) │ 100% (794 / 794) │
+ ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
+
+
+
+.. parsed-literal::
+
+ Output()
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. raw:: html
+
+
+
+
+
+
+
+
+.. parsed-literal::
+
+ 112933
+
+
+
+VAE Decoder
+~~~~~~~~~~~
+
+
+
+The VAE model has two parts, an encoder and a decoder. The encoder is
+used to convert the image into a low dimensional latent representation,
+which will serve as the input to the U-Net model. The decoder,
+conversely, transforms the latent representation back into an image.
+
+When running Text-to-Image pipeline, we will see that we only need the
+VAE decoder.
+
+.. code:: ipython3
+
+ vae_decoder = pipe.vae
+ vae_decoder.eval()
+
+
+ class VAEDecoderWrapper(torch.nn.Module):
+ def __init__(self, vae_decoder):
+ super().__init__()
+ self.vae = vae_decoder
+
+ def forward(self, latents):
+ return self.vae.decode(latents)
+
+
+ w_vae_decoder = VAEDecoderWrapper(vae_decoder)
+ inputs = torch.zeros((1, 4, 128, 128))
+
+ convert(w_vae_decoder, VAE_DECODER_OV_PATH, inputs, input_info=[1, 4, 128, 128])
+
+ del w_vae_decoder, vae_decoder
+ gc.collect()
+
+
+.. parsed-literal::
+
+ INFO:nncf:Statistics of the bitwidth distribution:
+ ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
+ │ Num bits (N) │ % all parameters (layers) │ % ratio-defining parameters (layers) │
+ ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
+ │ 8 │ 100% (40 / 40) │ 100% (40 / 40) │
+ ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
+
+
+
+.. parsed-literal::
+
+ Output()
+
+
+
+.. raw:: html
+
+
+
+
+
+
+.. raw:: html
+
+
+
+
+
+
+
+
+.. parsed-literal::
+
+ 6286
+
+
+
+Prepare Inference pipeline
+--------------------------
+
+
+
+In this example, we will reuse ``PhotoMakerStableDiffusionXLPipeline``
+pipeline to generate the image with OpenVINO, so each model’s object in
+this pipeline should be replaced with new OpenVINO model object.
+
+Select inference device for Stable Diffusion pipeline
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+.. code:: ipython3
+
+ import ipywidgets as widgets
+
+ core = ov.Core()
+
+ device = widgets.Dropdown(
+ options=core.available_devices + ["AUTO"],
+ value="CPU",
+ description="Device:",
+ disabled=False,
+ )
+
+ device
+
+
+
+
+.. parsed-literal::
+
+ Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
+
+
+
+Compile models and create their Wrappers for inference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+To access original PhotoMaker workflow, we have to create a new wrapper
+for each OpenVINO compiled model. For matching original pipeline, part
+of OpenVINO model wrapper’s attributes should be reused from original
+model objects and inference output must be converted from numpy to
+``torch.tensor``.
+
+
+
+.. code:: ipython3
+
+ compiled_id_encoder = core.compile_model(ID_ENCODER_OV_PATH, device.value)
+ compiled_unet = core.compile_model(UNET_OV_PATH, device.value)
+ compiled_text_encoder = core.compile_model(TEXT_ENCODER_OV_PATH, device.value)
+ compiled_text_encoder_2 = core.compile_model(TEXT_ENCODER_2_OV_PATH, device.value)
+ compiled_vae_decoder = core.compile_model(VAE_DECODER_OV_PATH, device.value)
+
+.. code:: ipython3
+
+ from collections import namedtuple
+
+
+ class OVIDEncoderWrapper(PhotoMakerIDEncoder):
+ dtype = torch.float32 # accessed in the original workflow
+
+ def __init__(self, id_encoder, orig_id_encoder):
+ super().__init__()
+ self.id_encoder = id_encoder
+ self.modules = orig_id_encoder.modules # accessed in the original workflow
+ self.config = orig_id_encoder.config # accessed in the original workflow
+
+ def __call__(
+ self,
+ *args,
+ ):
+ id_pixel_values, prompt_embeds, class_tokens_mask = args
+ inputs = {
+ "id_pixel_values": id_pixel_values,
+ "prompt_embeds": prompt_embeds,
+ "class_tokens_mask": class_tokens_mask,
+ }
+ output = self.id_encoder(inputs)[0]
+ return torch.from_numpy(output)
+
+.. code:: ipython3
+
+ class OVTextEncoderWrapper:
+ dtype = torch.float32 # accessed in the original workflow
+
+ def __init__(self, text_encoder, orig_text_encoder):
+ self.text_encoder = text_encoder
+ self.modules = orig_text_encoder.modules # accessed in the original workflow
+ self.config = orig_text_encoder.config # accessed in the original workflow
+
+ def __call__(self, input_ids, **kwargs):
+ inputs = {"input_ids": input_ids}
+ output = self.text_encoder(inputs)
+
+ hidden_states = []
+ hidden_states_len = len(output)
+ for i in range(1, hidden_states_len):
+ hidden_states.append(torch.from_numpy(output[i]))
+
+ BaseModelOutputWithPooling = namedtuple("BaseModelOutputWithPooling", "last_hidden_state hidden_states")
+ output = BaseModelOutputWithPooling(torch.from_numpy(output[0]), hidden_states)
+ return output
+
+.. code:: ipython3
+
+ class OVUnetWrapper:
+ def __init__(self, unet, unet_orig):
+ self.unet = unet
+ self.config = unet_orig.config # accessed in the original workflow
+ self.add_embedding = unet_orig.add_embedding # accessed in the original workflow
+
+ def __call__(self, *args, **kwargs):
+ latent_model_input, t = args
+ inputs = {
+ "sample": latent_model_input,
+ "timestep": t,
+ "encoder_hidden_states": kwargs["encoder_hidden_states"],
+ "text_embeds": kwargs["added_cond_kwargs"]["text_embeds"],
+ "time_ids": kwargs["added_cond_kwargs"]["time_ids"],
+ }
+
+ output = self.unet(inputs)
+
+ return [torch.from_numpy(output[0])]
+
+.. code:: ipython3
+
+ class OVVAEDecoderWrapper:
+ dtype = torch.float32 # accessed in the original workflow
+
+ def __init__(self, vae, vae_orig):
+ self.vae = vae
+ self.config = vae_orig.config # accessed in the original workflow
+
+ def decode(self, latents, return_dict=False):
+ output = self.vae(latents)[0]
+ output = torch.from_numpy(output)
+
+ return [output]
+
+Replace the PyTorch model objects in original pipeline with OpenVINO
+models
+
+.. code:: ipython3
+
+ pipe.id_encoder = OVIDEncoderWrapper(compiled_id_encoder, pipe.id_encoder)
+ pipe.unet = OVUnetWrapper(compiled_unet, pipe.unet)
+ pipe.text_encoder = OVTextEncoderWrapper(compiled_text_encoder, pipe.text_encoder)
+ pipe.text_encoder_2 = OVTextEncoderWrapper(compiled_text_encoder_2, pipe.text_encoder_2)
+ pipe.vae = OVVAEDecoderWrapper(compiled_vae_decoder, pipe.vae)
+
+Running Text-to-Image Generation with OpenVINO
+----------------------------------------------
+
+
+
+.. code:: ipython3
+
+ from diffusers.utils import load_image
+
+ prompt = "sci-fi, closeup portrait photo of a man img in Iron man suit, face"
+ negative_prompt = "(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth"
+ generator = torch.Generator("cpu").manual_seed(42)
+
+ input_id_images = []
+ original_image = load_image("./PhotoMaker/examples/newton_man/newton_0.jpg")
+ input_id_images.append(original_image)
+
+ ## Parameter setting
+ num_steps = 20
+ style_strength_ratio = 20
+ start_merge_step = int(float(style_strength_ratio) / 100 * num_steps)
+ if start_merge_step > 30:
+ start_merge_step = 30
+
+ images = pipe(
+ prompt=prompt,
+ input_id_images=input_id_images,
+ negative_prompt=negative_prompt,
+ num_images_per_prompt=1,
+ num_inference_steps=num_steps,
+ start_merge_step=start_merge_step,
+ generator=generator,
+ ).images
+
+
+
+.. parsed-literal::
+
+ 0%| | 0/20 [00:00 30:
+ start_merge_step = 30
+ result = pipe(
+ text_promt,
+ input_id_images=input_image,
+ negative_prompt=neg_prompt,
+ num_inference_steps=num_steps,
+ num_images_per_prompt=1,
+ start_merge_step=start_merge_step,
+ generator=torch.Generator().manual_seed(seed),
+ height=1024,
+ width=1024,
+ ).images[0]
+
+ return result
+
+
+ with gr.Blocks() as demo:
+ with gr.Column():
+ with gr.Row():
+ input_image = gr.Image(label="Your image", sources=["upload"], type="pil")
+ output_image = gr.Image(label="Generated Images", type="pil")
+ positive_input = gr.Textbox(label=f"Text prompt, Trigger words is '{trigger_word}'")
+ neg_input = gr.Textbox(label="Negative prompt")
+ with gr.Row():
+ seed_input = gr.Slider(0, 10_000_000, value=42, label="Seed")
+ steps_input = gr.Slider(label="Steps", value=10, minimum=5, maximum=50, step=1)
+ style_strength_ratio_input = gr.Slider(label="Style strength ratio", value=20, minimum=5, maximum=100, step=5)
+ btn = gr.Button()
+ btn.click(
+ generate_from_text,
+ [
+ positive_input,
+ input_image,
+ neg_input,
+ seed_input,
+ steps_input,
+ style_strength_ratio_input,
+ ],
+ output_image,
+ )
+ gr.Examples(
+ [
+ [prompt, negative_prompt],
+ [
+ "A woman img wearing a Christmas hat",
+ negative_prompt,
+ ],
+ [
+ "A man img in a helmet and vest riding a motorcycle",
+ negative_prompt,
+ ],
+ [
+ "photo of a middle-aged man img sitting on a plush leather couch, and watching television show",
+ negative_prompt,
+ ],
+ [
+ "photo of a skilled doctor img in a pristine white lab coat enjoying a delicious meal in a sophisticated dining room",
+ negative_prompt,
+ ],
+ [
+ "photo of superman img flying through a vibrant sunset sky, with his cape billowing in the wind",
+ negative_prompt,
+ ],
+ ],
+ [positive_input, neg_input],
+ )
+
+
+ demo.queue().launch()
+ # if you are launching remotely, specify server_name and server_port
+ # demo.launch(server_name='your server name', server_port='server port in int')
+ # Read more in the docs: https://gradio.app/docs/
+
+
+.. parsed-literal::
+
+ Running on local URL: http://127.0.0.1:7860
+
+ To create a public link, set `share=True` in `launch()`.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+.. code:: ipython3
+
+ demo.close()
+
+
+.. parsed-literal::
+
+ Closing server running on port: 7860
+
diff --git a/docs/notebooks/photo-maker-with-output_files/photo-maker-with-output_33_0.png b/docs/notebooks/photo-maker-with-output_files/photo-maker-with-output_33_0.png
new file mode 100644
index 00000000000..9ed4969e620
--- /dev/null
+++ b/docs/notebooks/photo-maker-with-output_files/photo-maker-with-output_33_0.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba2722c4b968ff96e1195ce80d01324804b30f97ba21a7edeb9e6d736bb3f507
+size 357391
diff --git a/docs/notebooks/pose-estimation-with-output.rst b/docs/notebooks/pose-estimation-with-output.rst
index cd612e90d93..2a20a6a0672 100644
--- a/docs/notebooks/pose-estimation-with-output.rst
+++ b/docs/notebooks/pose-estimation-with-output.rst
@@ -61,8 +61,6 @@ Imports
from numpy.lib.stride_tricks import as_strided
import openvino as ov
- from decoder import OpenPoseDecoder
-
# Fetch `notebook_utils` module
import requests
@@ -197,11 +195,6 @@ there is 1 input and 2 outputs: PAFs and keypoints heatmap.
-Processing
-----------
-
-
-
OpenPose Decoder
~~~~~~~~~~~~~~~~
@@ -219,6 +212,337 @@ which is available in the `demos
section `__
of Open Model Zoo.
+.. code:: ipython3
+
+ # code from https://github.com/openvinotoolkit/open_model_zoo/blob/9296a3712069e688fe64ea02367466122c8e8a3b/demos/common/python/models/open_pose.py#L135
+ class OpenPoseDecoder:
+ BODY_PARTS_KPT_IDS = (
+ (1, 2),
+ (1, 5),
+ (2, 3),
+ (3, 4),
+ (5, 6),
+ (6, 7),
+ (1, 8),
+ (8, 9),
+ (9, 10),
+ (1, 11),
+ (11, 12),
+ (12, 13),
+ (1, 0),
+ (0, 14),
+ (14, 16),
+ (0, 15),
+ (15, 17),
+ (2, 16),
+ (5, 17),
+ )
+ BODY_PARTS_PAF_IDS = (
+ 12,
+ 20,
+ 14,
+ 16,
+ 22,
+ 24,
+ 0,
+ 2,
+ 4,
+ 6,
+ 8,
+ 10,
+ 28,
+ 30,
+ 34,
+ 32,
+ 36,
+ 18,
+ 26,
+ )
+
+ def __init__(
+ self,
+ num_joints=18,
+ skeleton=BODY_PARTS_KPT_IDS,
+ paf_indices=BODY_PARTS_PAF_IDS,
+ max_points=100,
+ score_threshold=0.1,
+ min_paf_alignment_score=0.05,
+ delta=0.5,
+ ):
+ self.num_joints = num_joints
+ self.skeleton = skeleton
+ self.paf_indices = paf_indices
+ self.max_points = max_points
+ self.score_threshold = score_threshold
+ self.min_paf_alignment_score = min_paf_alignment_score
+ self.delta = delta
+
+ self.points_per_limb = 10
+ self.grid = np.arange(self.points_per_limb, dtype=np.float32).reshape(1, -1, 1)
+
+ def __call__(self, heatmaps, nms_heatmaps, pafs):
+ batch_size, _, h, w = heatmaps.shape
+ assert batch_size == 1, "Batch size of 1 only supported"
+
+ keypoints = self.extract_points(heatmaps, nms_heatmaps)
+ pafs = np.transpose(pafs, (0, 2, 3, 1))
+
+ if self.delta > 0:
+ for kpts in keypoints:
+ kpts[:, :2] += self.delta
+ np.clip(kpts[:, 0], 0, w - 1, out=kpts[:, 0])
+ np.clip(kpts[:, 1], 0, h - 1, out=kpts[:, 1])
+
+ pose_entries, keypoints = self.group_keypoints(keypoints, pafs, pose_entry_size=self.num_joints + 2)
+ poses, scores = self.convert_to_coco_format(pose_entries, keypoints)
+ if len(poses) > 0:
+ poses = np.asarray(poses, dtype=np.float32)
+ poses = poses.reshape((poses.shape[0], -1, 3))
+ else:
+ poses = np.empty((0, 17, 3), dtype=np.float32)
+ scores = np.empty(0, dtype=np.float32)
+
+ return poses, scores
+
+ def extract_points(self, heatmaps, nms_heatmaps):
+ batch_size, channels_num, h, w = heatmaps.shape
+ assert batch_size == 1, "Batch size of 1 only supported"
+ assert channels_num >= self.num_joints
+
+ xs, ys, scores = self.top_k(nms_heatmaps)
+ masks = scores > self.score_threshold
+ all_keypoints = []
+ keypoint_id = 0
+ for k in range(self.num_joints):
+ # Filter low-score points.
+ mask = masks[0, k]
+ x = xs[0, k][mask].ravel()
+ y = ys[0, k][mask].ravel()
+ score = scores[0, k][mask].ravel()
+ n = len(x)
+ if n == 0:
+ all_keypoints.append(np.empty((0, 4), dtype=np.float32))
+ continue
+ # Apply quarter offset to improve localization accuracy.
+ x, y = self.refine(heatmaps[0, k], x, y)
+ np.clip(x, 0, w - 1, out=x)
+ np.clip(y, 0, h - 1, out=y)
+ # Pack resulting points.
+ keypoints = np.empty((n, 4), dtype=np.float32)
+ keypoints[:, 0] = x
+ keypoints[:, 1] = y
+ keypoints[:, 2] = score
+ keypoints[:, 3] = np.arange(keypoint_id, keypoint_id + n)
+ keypoint_id += n
+ all_keypoints.append(keypoints)
+ return all_keypoints
+
+ def top_k(self, heatmaps):
+ N, K, _, W = heatmaps.shape
+ heatmaps = heatmaps.reshape(N, K, -1)
+ # Get positions with top scores.
+ ind = heatmaps.argpartition(-self.max_points, axis=2)[:, :, -self.max_points :]
+ scores = np.take_along_axis(heatmaps, ind, axis=2)
+ # Keep top scores sorted.
+ subind = np.argsort(-scores, axis=2)
+ ind = np.take_along_axis(ind, subind, axis=2)
+ scores = np.take_along_axis(scores, subind, axis=2)
+ y, x = np.divmod(ind, W)
+ return x, y, scores
+
+ @staticmethod
+ def refine(heatmap, x, y):
+ h, w = heatmap.shape[-2:]
+ valid = np.logical_and(np.logical_and(x > 0, x < w - 1), np.logical_and(y > 0, y < h - 1))
+ xx = x[valid]
+ yy = y[valid]
+ dx = np.sign(heatmap[yy, xx + 1] - heatmap[yy, xx - 1], dtype=np.float32) * 0.25
+ dy = np.sign(heatmap[yy + 1, xx] - heatmap[yy - 1, xx], dtype=np.float32) * 0.25
+ x = x.astype(np.float32)
+ y = y.astype(np.float32)
+ x[valid] += dx
+ y[valid] += dy
+ return x, y
+
+ @staticmethod
+ def is_disjoint(pose_a, pose_b):
+ pose_a = pose_a[:-2]
+ pose_b = pose_b[:-2]
+ return np.all(np.logical_or.reduce((pose_a == pose_b, pose_a < 0, pose_b < 0)))
+
+ def update_poses(
+ self,
+ kpt_a_id,
+ kpt_b_id,
+ all_keypoints,
+ connections,
+ pose_entries,
+ pose_entry_size,
+ ):
+ for connection in connections:
+ pose_a_idx = -1
+ pose_b_idx = -1
+ for j, pose in enumerate(pose_entries):
+ if pose[kpt_a_id] == connection[0]:
+ pose_a_idx = j
+ if pose[kpt_b_id] == connection[1]:
+ pose_b_idx = j
+ if pose_a_idx < 0 and pose_b_idx < 0:
+ # Create new pose entry.
+ pose_entry = np.full(pose_entry_size, -1, dtype=np.float32)
+ pose_entry[kpt_a_id] = connection[0]
+ pose_entry[kpt_b_id] = connection[1]
+ pose_entry[-1] = 2
+ pose_entry[-2] = np.sum(all_keypoints[connection[0:2], 2]) + connection[2]
+ pose_entries.append(pose_entry)
+ elif pose_a_idx >= 0 and pose_b_idx >= 0 and pose_a_idx != pose_b_idx:
+ # Merge two poses are disjoint merge them, otherwise ignore connection.
+ pose_a = pose_entries[pose_a_idx]
+ pose_b = pose_entries[pose_b_idx]
+ if self.is_disjoint(pose_a, pose_b):
+ pose_a += pose_b
+ pose_a[:-2] += 1
+ pose_a[-2] += connection[2]
+ del pose_entries[pose_b_idx]
+ elif pose_a_idx >= 0 and pose_b_idx >= 0:
+ # Adjust score of a pose.
+ pose_entries[pose_a_idx][-2] += connection[2]
+ elif pose_a_idx >= 0:
+ # Add a new limb into pose.
+ pose = pose_entries[pose_a_idx]
+ if pose[kpt_b_id] < 0:
+ pose[-2] += all_keypoints[connection[1], 2]
+ pose[kpt_b_id] = connection[1]
+ pose[-2] += connection[2]
+ pose[-1] += 1
+ elif pose_b_idx >= 0:
+ # Add a new limb into pose.
+ pose = pose_entries[pose_b_idx]
+ if pose[kpt_a_id] < 0:
+ pose[-2] += all_keypoints[connection[0], 2]
+ pose[kpt_a_id] = connection[0]
+ pose[-2] += connection[2]
+ pose[-1] += 1
+ return pose_entries
+
+ @staticmethod
+ def connections_nms(a_idx, b_idx, affinity_scores):
+ # From all retrieved connections that share starting/ending keypoints leave only the top-scoring ones.
+ order = affinity_scores.argsort()[::-1]
+ affinity_scores = affinity_scores[order]
+ a_idx = a_idx[order]
+ b_idx = b_idx[order]
+ idx = []
+ has_kpt_a = set()
+ has_kpt_b = set()
+ for t, (i, j) in enumerate(zip(a_idx, b_idx)):
+ if i not in has_kpt_a and j not in has_kpt_b:
+ idx.append(t)
+ has_kpt_a.add(i)
+ has_kpt_b.add(j)
+ idx = np.asarray(idx, dtype=np.int32)
+ return a_idx[idx], b_idx[idx], affinity_scores[idx]
+
+ def group_keypoints(self, all_keypoints_by_type, pafs, pose_entry_size=20):
+ all_keypoints = np.concatenate(all_keypoints_by_type, axis=0)
+ pose_entries = []
+ # For every limb.
+ for part_id, paf_channel in enumerate(self.paf_indices):
+ kpt_a_id, kpt_b_id = self.skeleton[part_id]
+ kpts_a = all_keypoints_by_type[kpt_a_id]
+ kpts_b = all_keypoints_by_type[kpt_b_id]
+ n = len(kpts_a)
+ m = len(kpts_b)
+ if n == 0 or m == 0:
+ continue
+
+ # Get vectors between all pairs of keypoints, i.e. candidate limb vectors.
+ a = kpts_a[:, :2]
+ a = np.broadcast_to(a[None], (m, n, 2))
+ b = kpts_b[:, :2]
+ vec_raw = (b[:, None, :] - a).reshape(-1, 1, 2)
+
+ # Sample points along every candidate limb vector.
+ steps = 1 / (self.points_per_limb - 1) * vec_raw
+ points = steps * self.grid + a.reshape(-1, 1, 2)
+ points = points.round().astype(dtype=np.int32)
+ x = points[..., 0].ravel()
+ y = points[..., 1].ravel()
+
+ # Compute affinity score between candidate limb vectors and part affinity field.
+ part_pafs = pafs[0, :, :, paf_channel : paf_channel + 2]
+ field = part_pafs[y, x].reshape(-1, self.points_per_limb, 2)
+ vec_norm = np.linalg.norm(vec_raw, ord=2, axis=-1, keepdims=True)
+ vec = vec_raw / (vec_norm + 1e-6)
+ affinity_scores = (field * vec).sum(-1).reshape(-1, self.points_per_limb)
+ valid_affinity_scores = affinity_scores > self.min_paf_alignment_score
+ valid_num = valid_affinity_scores.sum(1)
+ affinity_scores = (affinity_scores * valid_affinity_scores).sum(1) / (valid_num + 1e-6)
+ success_ratio = valid_num / self.points_per_limb
+
+ # Get a list of limbs according to the obtained affinity score.
+ valid_limbs = np.where(np.logical_and(affinity_scores > 0, success_ratio > 0.8))[0]
+ if len(valid_limbs) == 0:
+ continue
+ b_idx, a_idx = np.divmod(valid_limbs, n)
+ affinity_scores = affinity_scores[valid_limbs]
+
+ # Suppress incompatible connections.
+ a_idx, b_idx, affinity_scores = self.connections_nms(a_idx, b_idx, affinity_scores)
+ connections = list(
+ zip(
+ kpts_a[a_idx, 3].astype(np.int32),
+ kpts_b[b_idx, 3].astype(np.int32),
+ affinity_scores,
+ )
+ )
+ if len(connections) == 0:
+ continue
+
+ # Update poses with new connections.
+ pose_entries = self.update_poses(
+ kpt_a_id,
+ kpt_b_id,
+ all_keypoints,
+ connections,
+ pose_entries,
+ pose_entry_size,
+ )
+
+ # Remove poses with not enough points.
+ pose_entries = np.asarray(pose_entries, dtype=np.float32).reshape(-1, pose_entry_size)
+ pose_entries = pose_entries[pose_entries[:, -1] >= 3]
+ return pose_entries, all_keypoints
+
+ @staticmethod
+ def convert_to_coco_format(pose_entries, all_keypoints):
+ num_joints = 17
+ coco_keypoints = []
+ scores = []
+ for pose in pose_entries:
+ if len(pose) == 0:
+ continue
+ keypoints = np.zeros(num_joints * 3)
+ reorder_map = [0, -1, 6, 8, 10, 5, 7, 9, 12, 14, 16, 11, 13, 15, 2, 1, 4, 3]
+ person_score = pose[-2]
+ for keypoint_id, target_id in zip(pose[:-2], reorder_map):
+ if target_id < 0:
+ continue
+ cx, cy, score = 0, 0, 0 # keypoint not found
+ if keypoint_id != -1:
+ cx, cy, score = all_keypoints[int(keypoint_id), 0:3]
+ keypoints[target_id * 3 + 0] = cx
+ keypoints[target_id * 3 + 1] = cy
+ keypoints[target_id * 3 + 2] = score
+ coco_keypoints.append(keypoints)
+ scores.append(person_score * max(0, (pose[-1] - 1))) # -1 for 'neck'
+ return np.asarray(coco_keypoints), np.asarray(scores)
+
+Processing
+----------
+
+
+
.. code:: ipython3
decoder = OpenPoseDecoder()
@@ -525,7 +849,7 @@ Run the pose estimation:
-.. image:: pose-estimation-with-output_files/pose-estimation-with-output_20_0.png
+.. image:: pose-estimation-with-output_files/pose-estimation-with-output_22_0.png
.. parsed-literal::
diff --git a/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_20_0.png b/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_20_0.png
deleted file mode 100644
index 2614140c287..00000000000
--- a/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_20_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dc169ae2255a627b398a42161a5ebafb0d22d4a644bfc15cfa8b050471910991
-size 108138
diff --git a/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_22_0.png b/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_22_0.png
new file mode 100644
index 00000000000..970e77fef37
--- /dev/null
+++ b/docs/notebooks/pose-estimation-with-output_files/pose-estimation-with-output_22_0.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a74b5a0589bb13eb1e3e6be207a0682193cf5b0a3b721075a78c5145bd207cb
+size 108163
diff --git a/docs/notebooks/pyannote-speaker-diarization-with-output.rst b/docs/notebooks/pyannote-speaker-diarization-with-output.rst
index aeee17d8f1f..8033d0e223d 100644
--- a/docs/notebooks/pyannote-speaker-diarization-with-output.rst
+++ b/docs/notebooks/pyannote-speaker-diarization-with-output.rst
@@ -65,25 +65,13 @@ Prerequisites
.. parsed-literal::
WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
googleapis-common-protos 1.63.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
onnx 1.16.0 requires protobuf>=3.20.2, but you have protobuf 3.20.1 which is incompatible.
paddlepaddle 2.6.1 requires protobuf>=3.20.2; platform_system != "Windows", but you have protobuf 3.20.1 which is incompatible.
tensorflow 2.12.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 3.20.1 which is incompatible.
tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 3.20.1 which is incompatible.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -168,19 +156,11 @@ hub `__.
.. parsed-literal::
- 2024-04-18 00:26:15.129545: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:26:15.165135: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 00:55:33.958421: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 00:55:33.992823: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:26:15.736692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/pyannote/audio/core/io.py:42: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
+ 2024-05-07 00:55:34.728268: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/pyannote/audio/core/io.py:42: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
torchaudio.set_audio_backend("soundfile")
@@ -271,7 +251,7 @@ pipeline
.. parsed-literal::
- Diarization pipeline took 16.70 s
+ Diarization pipeline took 17.11 s
The result of running the pipeline can be represented as a diagram
@@ -434,10 +414,6 @@ Run speaker diarization with OpenVINO
.. parsed-literal::
Model is not converging. Current: 16444.281598619917 is not greater than 16444.330820454463. Delta is -0.04922183454618789
-
-
-.. parsed-literal::
-
Model is not converging. Current: 16444.281598619917 is not greater than 16444.330820454463. Delta is -0.04922183454618789
@@ -448,7 +424,7 @@ Run speaker diarization with OpenVINO
.. parsed-literal::
- Diarization pipeline took 16.17 s
+ Diarization pipeline took 16.62 s
.. code:: ipython3
diff --git a/docs/notebooks/pytorch-onnx-to-openvino-with-output.rst b/docs/notebooks/pytorch-onnx-to-openvino-with-output.rst
index 470ad982c3a..3a08b651645 100644
--- a/docs/notebooks/pytorch-onnx-to-openvino-with-output.rst
+++ b/docs/notebooks/pytorch-onnx-to-openvino-with-output.rst
@@ -75,20 +75,12 @@ Table of contents:
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
googleapis-common-protos 1.63.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5, but you have protobuf 5.26.1 which is incompatible.
pytorch-lightning 1.6.5 requires protobuf<=3.20.1, but you have protobuf 5.26.1 which is incompatible.
tensorflow 2.12.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 5.26.1 which is incompatible.
tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 5.26.1 which is incompatible.
tf2onnx 1.16.1 requires protobuf~=3.20, but you have protobuf 5.26.1 which is incompatible.
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -574,17 +566,9 @@ performance.
.. parsed-literal::
- PyTorch model on CPU: 0.040 seconds per image, FPS: 25.21
-
-
-.. parsed-literal::
-
- ONNX model in OpenVINO Runtime/AUTO: 0.018 seconds per image, FPS: 54.74
-
-
-.. parsed-literal::
-
- OpenVINO IR model in OpenVINO Runtime/AUTO: 0.028 seconds per image, FPS: 35.55
+ PyTorch model on CPU: 0.039 seconds per image, FPS: 25.93
+ ONNX model in OpenVINO Runtime/AUTO: 0.018 seconds per image, FPS: 54.11
+ OpenVINO IR model in OpenVINO Runtime/AUTO: 0.028 seconds per image, FPS: 35.60
**Show Device Information**
diff --git a/docs/notebooks/pytorch-post-training-quantization-nncf-with-output.rst b/docs/notebooks/pytorch-post-training-quantization-nncf-with-output.rst
index 7849e520edb..f46124855f1 100644
--- a/docs/notebooks/pytorch-post-training-quantization-nncf-with-output.rst
+++ b/docs/notebooks/pytorch-post-training-quantization-nncf-with-output.rst
@@ -65,20 +65,8 @@ Preparations
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -162,7 +150,7 @@ Settings
.. parsed-literal::
- PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
+ PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/pytorch-post-training-quantization-nncf/model/resnet50_fp32.pth')
@@ -452,47 +440,15 @@ I. Evaluate the loaded model
.. parsed-literal::
- Test: [ 0/79] Time 0.261 (0.261) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
-
-
-.. parsed-literal::
-
- Test: [10/79] Time 0.234 (0.238) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
-
-
-.. parsed-literal::
-
- Test: [20/79] Time 0.267 (0.239) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
-
-
-.. parsed-literal::
-
- Test: [30/79] Time 0.259 (0.239) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
-
-
-.. parsed-literal::
-
- Test: [40/79] Time 0.237 (0.239) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
-
-
-.. parsed-literal::
-
- Test: [50/79] Time 0.236 (0.240) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
-
-
-.. parsed-literal::
-
- Test: [60/79] Time 0.248 (0.240) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
-
-
-.. parsed-literal::
-
- Test: [70/79] Time 0.248 (0.242) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
-
-
-.. parsed-literal::
-
- * Acc@1 60.740 Acc@5 83.960 Total time: 18.845
+ Test: [ 0/79] Time 0.249 (0.249) Acc@1 81.25 (81.25) Acc@5 92.19 (92.19)
+ Test: [10/79] Time 0.223 (0.231) Acc@1 56.25 (66.97) Acc@5 86.72 (87.50)
+ Test: [20/79] Time 0.224 (0.227) Acc@1 67.97 (64.29) Acc@5 85.16 (87.35)
+ Test: [30/79] Time 0.223 (0.232) Acc@1 53.12 (62.37) Acc@5 77.34 (85.33)
+ Test: [40/79] Time 0.225 (0.232) Acc@1 67.19 (60.86) Acc@5 90.62 (84.51)
+ Test: [50/79] Time 0.224 (0.231) Acc@1 60.16 (60.80) Acc@5 88.28 (84.42)
+ Test: [60/79] Time 0.214 (0.231) Acc@1 66.41 (60.46) Acc@5 86.72 (83.79)
+ Test: [70/79] Time 0.218 (0.230) Acc@1 52.34 (60.21) Acc@5 80.47 (83.33)
+ * Acc@1 60.740 Acc@5 83.960 Total time: 17.917
Test accuracy of FP32 model: 60.740
@@ -535,19 +491,15 @@ Guide =1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -167,7 +155,7 @@ models will be stored.
.. parsed-literal::
- PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/pytorch-quantization-aware-training/model/resnet18_fp32.pth')
+ PosixPath('/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/pytorch-quantization-aware-training/model/resnet18_fp32.pth')
@@ -472,9 +460,9 @@ section at the top of this notebook.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
warnings.warn(msg)
@@ -584,19 +572,15 @@ about supported parameters can be found on this
.. parsed-literal::
- 2024-04-18 00:33:36.062903: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:33:36.099858: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 01:02:39.722072: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 01:02:39.758422: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
+ 2024-05-07 01:02:40.294932: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
.. parsed-literal::
- 2024-04-18 00:33:36.657984: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- WARNING:nncf:NNCF provides best results with torch==2.1.2, while current torch version is 2.2.2+cpu. If you encounter issues, consider switching to torch==2.1.2
+ WARNING:nncf:NNCF provides best results with torch==2.2.*, while current torch version is 2.3.0+cpu. If you encounter issues, consider switching to torch==2.2.*
@@ -623,10 +607,6 @@ about supported parameters can be found on this
.. parsed-literal::
INFO:nncf:Compiling and loading torch extension: quantized_functions_cpu...
-
-
-.. parsed-literal::
-
INFO:nncf:Finished loading torch extension: quantized_functions_cpu
@@ -664,46 +644,14 @@ demonstrated here.
.. parsed-literal::
- Test: [ 0/79] Time 0.183 (0.183) Loss 1.005 (1.005) Acc@1 78.91 (78.91) Acc@5 88.28 (88.28)
-
-
-.. parsed-literal::
-
- Test: [10/79] Time 0.144 (0.153) Loss 1.992 (1.625) Acc@1 44.53 (60.37) Acc@5 79.69 (83.66)
-
-
-.. parsed-literal::
-
- Test: [20/79] Time 0.145 (0.149) Loss 1.814 (1.705) Acc@1 60.94 (58.04) Acc@5 80.47 (82.66)
-
-
-.. parsed-literal::
-
- Test: [30/79] Time 0.146 (0.148) Loss 2.287 (1.795) Acc@1 50.78 (56.48) Acc@5 68.75 (80.97)
-
-
-.. parsed-literal::
-
- Test: [40/79] Time 0.146 (0.147) Loss 1.615 (1.832) Acc@1 60.94 (55.43) Acc@5 82.81 (80.43)
-
-
-.. parsed-literal::
-
- Test: [50/79] Time 0.144 (0.147) Loss 1.952 (1.833) Acc@1 57.03 (55.51) Acc@5 75.00 (80.16)
-
-
-.. parsed-literal::
-
- Test: [60/79] Time 0.143 (0.146) Loss 1.794 (1.856) Acc@1 57.03 (55.16) Acc@5 84.38 (79.84)
-
-
-.. parsed-literal::
-
- Test: [70/79] Time 0.144 (0.146) Loss 2.371 (1.889) Acc@1 46.88 (54.68) Acc@5 74.22 (79.14)
-
-
-.. parsed-literal::
-
+ Test: [ 0/79] Time 0.202 (0.202) Loss 1.005 (1.005) Acc@1 78.91 (78.91) Acc@5 88.28 (88.28)
+ Test: [10/79] Time 0.164 (0.169) Loss 1.992 (1.625) Acc@1 44.53 (60.37) Acc@5 79.69 (83.66)
+ Test: [20/79] Time 0.163 (0.167) Loss 1.814 (1.705) Acc@1 60.94 (58.04) Acc@5 80.47 (82.66)
+ Test: [30/79] Time 0.163 (0.166) Loss 2.287 (1.795) Acc@1 50.78 (56.48) Acc@5 68.75 (80.97)
+ Test: [40/79] Time 0.171 (0.167) Loss 1.615 (1.832) Acc@1 60.94 (55.43) Acc@5 82.81 (80.43)
+ Test: [50/79] Time 0.165 (0.166) Loss 1.952 (1.833) Acc@1 57.03 (55.51) Acc@5 75.00 (80.16)
+ Test: [60/79] Time 0.197 (0.167) Loss 1.794 (1.856) Acc@1 57.03 (55.16) Acc@5 84.38 (79.84)
+ Test: [70/79] Time 0.192 (0.169) Loss 2.371 (1.889) Acc@1 46.88 (54.68) Acc@5 74.22 (79.14)
* Acc@1 55.040 Acc@5 79.730
Accuracy of initialized INT8 model: 55.040
@@ -736,129 +684,33 @@ training pipeline are required. Here is a simple example.
.. parsed-literal::
- Epoch:[0][ 0/782] Time 0.410 (0.410) Loss 0.917 (0.917) Acc@1 76.56 (76.56) Acc@5 93.75 (93.75)
-
-
-.. parsed-literal::
-
- Epoch:[0][ 50/782] Time 0.363 (0.366) Loss 0.625 (0.812) Acc@1 87.50 (80.27) Acc@5 96.88 (93.92)
-
-
-.. parsed-literal::
-
- Epoch:[0][100/782] Time 0.361 (0.366) Loss 0.764 (0.807) Acc@1 79.69 (80.37) Acc@5 94.53 (94.17)
-
-
-.. parsed-literal::
-
- Epoch:[0][150/782] Time 0.363 (0.366) Loss 0.863 (0.799) Acc@1 82.81 (80.53) Acc@5 92.97 (94.25)
-
-
-.. parsed-literal::
-
- Epoch:[0][200/782] Time 0.365 (0.367) Loss 0.581 (0.787) Acc@1 85.16 (80.80) Acc@5 97.66 (94.34)
-
-
-.. parsed-literal::
-
- Epoch:[0][250/782] Time 0.363 (0.367) Loss 0.722 (0.782) Acc@1 82.81 (80.88) Acc@5 93.75 (94.42)
-
-
-.. parsed-literal::
-
- Epoch:[0][300/782] Time 0.365 (0.367) Loss 0.737 (0.777) Acc@1 78.91 (81.01) Acc@5 93.75 (94.41)
-
-
-.. parsed-literal::
-
- Epoch:[0][350/782] Time 0.366 (0.367) Loss 0.819 (0.767) Acc@1 80.47 (81.29) Acc@5 92.97 (94.53)
-
-
-.. parsed-literal::
-
- Epoch:[0][400/782] Time 0.368 (0.366) Loss 0.787 (0.767) Acc@1 80.47 (81.35) Acc@5 94.53 (94.53)
-
-
-.. parsed-literal::
-
- Epoch:[0][450/782] Time 0.364 (0.366) Loss 0.726 (0.763) Acc@1 82.03 (81.48) Acc@5 96.88 (94.55)
-
-
-.. parsed-literal::
-
- Epoch:[0][500/782] Time 0.365 (0.366) Loss 0.727 (0.760) Acc@1 82.03 (81.54) Acc@5 94.53 (94.58)
-
-
-.. parsed-literal::
-
- Epoch:[0][550/782] Time 0.365 (0.366) Loss 0.781 (0.758) Acc@1 82.81 (81.58) Acc@5 95.31 (94.59)
-
-
-.. parsed-literal::
-
- Epoch:[0][600/782] Time 0.362 (0.366) Loss 0.721 (0.756) Acc@1 80.47 (81.63) Acc@5 97.66 (94.61)
-
-
-.. parsed-literal::
-
- Epoch:[0][650/782] Time 0.365 (0.366) Loss 0.922 (0.755) Acc@1 76.56 (81.64) Acc@5 92.97 (94.63)
-
-
-.. parsed-literal::
-
- Epoch:[0][700/782] Time 0.366 (0.366) Loss 0.651 (0.753) Acc@1 83.59 (81.68) Acc@5 92.97 (94.63)
-
-
-.. parsed-literal::
-
- Epoch:[0][750/782] Time 0.369 (0.366) Loss 0.781 (0.751) Acc@1 80.47 (81.70) Acc@5 95.31 (94.66)
-
-
-.. parsed-literal::
-
- Test: [ 0/79] Time 0.146 (0.146) Loss 1.092 (1.092) Acc@1 73.44 (73.44) Acc@5 86.72 (86.72)
-
-
-.. parsed-literal::
-
- Test: [10/79] Time 0.145 (0.147) Loss 1.826 (1.522) Acc@1 49.22 (62.78) Acc@5 81.25 (84.23)
-
-
-.. parsed-literal::
-
- Test: [20/79] Time 0.147 (0.147) Loss 1.531 (1.594) Acc@1 64.84 (60.83) Acc@5 82.03 (83.85)
-
-
-.. parsed-literal::
-
- Test: [30/79] Time 0.148 (0.147) Loss 2.059 (1.690) Acc@1 57.03 (59.22) Acc@5 71.09 (82.26)
-
-
-.. parsed-literal::
-
- Test: [40/79] Time 0.147 (0.147) Loss 1.516 (1.744) Acc@1 64.06 (57.91) Acc@5 85.16 (81.46)
-
-
-.. parsed-literal::
-
- Test: [50/79] Time 0.147 (0.147) Loss 1.922 (1.750) Acc@1 53.12 (57.69) Acc@5 76.56 (81.14)
-
-
-.. parsed-literal::
-
- Test: [60/79] Time 0.146 (0.147) Loss 1.594 (1.785) Acc@1 65.62 (57.17) Acc@5 84.38 (80.60)
-
-
-.. parsed-literal::
-
- Test: [70/79] Time 0.141 (0.147) Loss 2.460 (1.811) Acc@1 46.09 (56.75) Acc@5 74.22 (80.08)
-
-
-.. parsed-literal::
-
- * Acc@1 57.180 Acc@5 80.680
- Accuracy of tuned INT8 model: 57.180
- Accuracy drop of tuned INT8 model over pre-trained FP32 model: -1.660
+ Epoch:[0][ 0/782] Time 0.416 (0.416) Loss 0.917 (0.917) Acc@1 76.56 (76.56) Acc@5 93.75 (93.75)
+ Epoch:[0][ 50/782] Time 0.369 (0.369) Loss 0.628 (0.812) Acc@1 87.50 (80.33) Acc@5 96.09 (93.92)
+ Epoch:[0][100/782] Time 0.364 (0.368) Loss 0.759 (0.806) Acc@1 79.69 (80.55) Acc@5 93.75 (94.12)
+ Epoch:[0][150/782] Time 0.363 (0.368) Loss 0.865 (0.799) Acc@1 82.81 (80.71) Acc@5 92.97 (94.18)
+ Epoch:[0][200/782] Time 0.363 (0.368) Loss 0.581 (0.787) Acc@1 86.72 (80.96) Acc@5 97.66 (94.32)
+ Epoch:[0][250/782] Time 0.363 (0.368) Loss 0.720 (0.782) Acc@1 83.59 (81.00) Acc@5 93.75 (94.42)
+ Epoch:[0][300/782] Time 0.363 (0.368) Loss 0.739 (0.777) Acc@1 78.91 (81.11) Acc@5 93.75 (94.40)
+ Epoch:[0][350/782] Time 0.364 (0.368) Loss 0.819 (0.767) Acc@1 78.12 (81.36) Acc@5 92.97 (94.52)
+ Epoch:[0][400/782] Time 0.375 (0.368) Loss 0.787 (0.766) Acc@1 80.47 (81.42) Acc@5 94.53 (94.51)
+ Epoch:[0][450/782] Time 0.366 (0.368) Loss 0.733 (0.763) Acc@1 82.03 (81.55) Acc@5 96.88 (94.54)
+ Epoch:[0][500/782] Time 0.368 (0.368) Loss 0.728 (0.760) Acc@1 82.81 (81.59) Acc@5 94.53 (94.57)
+ Epoch:[0][550/782] Time 0.371 (0.368) Loss 0.777 (0.758) Acc@1 83.59 (81.63) Acc@5 95.31 (94.59)
+ Epoch:[0][600/782] Time 0.372 (0.368) Loss 0.725 (0.756) Acc@1 80.47 (81.67) Acc@5 97.66 (94.60)
+ Epoch:[0][650/782] Time 0.363 (0.368) Loss 0.920 (0.755) Acc@1 76.56 (81.66) Acc@5 92.97 (94.62)
+ Epoch:[0][700/782] Time 0.365 (0.369) Loss 0.648 (0.753) Acc@1 84.38 (81.70) Acc@5 92.97 (94.63)
+ Epoch:[0][750/782] Time 0.366 (0.368) Loss 0.782 (0.750) Acc@1 80.47 (81.71) Acc@5 94.53 (94.66)
+ Test: [ 0/79] Time 0.145 (0.145) Loss 1.094 (1.094) Acc@1 76.56 (76.56) Acc@5 86.72 (86.72)
+ Test: [10/79] Time 0.143 (0.144) Loss 1.848 (1.527) Acc@1 49.22 (62.93) Acc@5 80.47 (84.30)
+ Test: [20/79] Time 0.143 (0.144) Loss 1.540 (1.597) Acc@1 64.06 (60.64) Acc@5 81.25 (83.82)
+ Test: [30/79] Time 0.146 (0.144) Loss 2.052 (1.692) Acc@1 56.25 (59.20) Acc@5 71.88 (82.21)
+ Test: [40/79] Time 0.143 (0.144) Loss 1.515 (1.745) Acc@1 64.06 (57.79) Acc@5 85.16 (81.44)
+ Test: [50/79] Time 0.143 (0.144) Loss 1.915 (1.751) Acc@1 53.91 (57.60) Acc@5 77.34 (81.14)
+ Test: [60/79] Time 0.143 (0.144) Loss 1.585 (1.786) Acc@1 67.19 (57.01) Acc@5 85.16 (80.69)
+ Test: [70/79] Time 0.142 (0.144) Loss 2.454 (1.812) Acc@1 44.53 (56.57) Acc@5 74.22 (80.24)
+ * Acc@1 56.970 Acc@5 80.830
+ Accuracy of tuned INT8 model: 56.970
+ Accuracy drop of tuned INT8 model over pre-trained FP32 model: -1.450
Export INT8 Model to OpenVINO IR
@@ -880,10 +732,6 @@ Export INT8 Model to OpenVINO IR
.. parsed-literal::
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
-
-
-.. parsed-literal::
-
INT8 model exported to model/resnet18_int8.xml.
@@ -952,17 +800,9 @@ throughput (frames per second) values.
.. parsed-literal::
Benchmark FP32 model (IR)
-
-
-.. parsed-literal::
-
- [ INFO ] Throughput: 2928.21 FPS
+ [ INFO ] Throughput: 2943.57 FPS
Benchmark INT8 model (IR)
-
-
-.. parsed-literal::
-
- [ INFO ] Throughput: 11678.74 FPS
+ [ INFO ] Throughput: 11901.74 FPS
Show Device Information for reference.
diff --git a/docs/notebooks/pytorch-to-openvino-with-output.rst b/docs/notebooks/pytorch-to-openvino-with-output.rst
index 12c29bb7635..20697dee1c5 100644
--- a/docs/notebooks/pytorch-to-openvino-with-output.rst
+++ b/docs/notebooks/pytorch-to-openvino-with-output.rst
@@ -90,10 +90,6 @@ Install notebook dependencies
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -255,7 +251,7 @@ Benchmark PyTorch Model Inference
.. parsed-literal::
- 16.6 ms ± 93.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 16.9 ms ± 130 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model to OpenVINO Intermediate Representation
@@ -414,7 +410,7 @@ Benchmark OpenVINO Model Inference
.. parsed-literal::
- 3.17 ms ± 17.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 3.16 ms ± 11 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Model with Static Input Shape
@@ -544,7 +540,7 @@ Benchmark OpenVINO Model Inference with Static Input Shape
.. parsed-literal::
- 2.88 ms ± 28 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 2.95 ms ± 6.16 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert TorchScript Model to OpenVINO Intermediate Representation
@@ -639,7 +635,7 @@ Benchmark Scripted Model Inference
.. parsed-literal::
- 17.5 ms ± 7.56 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
+ 14.1 ms ± 67.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Scripted Model to OpenVINO Intermediate Representation
@@ -698,7 +694,7 @@ Benchmark OpenVINO Model Inference Converted From Scripted Model
.. parsed-literal::
- 3.18 ms ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 3.27 ms ± 6.74 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Traced Model
@@ -774,7 +770,7 @@ Benchmark Traced Model Inference
.. parsed-literal::
- 13.7 ms ± 203 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 14.7 ms ± 193 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Convert PyTorch Traced Model to OpenVINO Intermediate Representation
@@ -833,5 +829,5 @@ Benchmark OpenVINO Model Inference Converted From Traced Model
.. parsed-literal::
- 3.24 ms ± 8.88 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+ 3.4 ms ± 3.83 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
diff --git a/docs/notebooks/question-answering-with-output.rst b/docs/notebooks/question-answering-with-output.rst
deleted file mode 100644
index e568322bf3f..00000000000
--- a/docs/notebooks/question-answering-with-output.rst
+++ /dev/null
@@ -1,637 +0,0 @@
-Interactive question answering with OpenVINO™
-=============================================
-
-This demo shows interactive question answering with OpenVINO, using
-`small BERT-large-like
-model `__
-distilled and quantized to ``INT8`` on SQuAD v1.1 training set from
-larger BERT-large model. The model comes from `Open Model
-Zoo `__. Final part
-of this notebook provides live inference results from your inputs.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Imports <#imports>`__
-- `The model <#the-model>`__
-
- - `Download the model <#download-the-model>`__
- - `Load the model <#load-the-model>`__
-
- - `Select inference device <#select-inference-device>`__
-
-- `Processing <#processing>`__
-
- - `Preprocessing <#preprocessing>`__
- - `Postprocessing <#postprocessing>`__
- - `Main Processing Function <#main-processing-function>`__
-
-- `Run <#run>`__
-
- - `Run on local paragraphs <#run-on-local-paragraphs>`__
- - `Run on websites <#run-on-websites>`__
-
-.. code:: ipython3
-
- %pip install -q "openvino>=2023.1.0" tqdm
-
-
-.. parsed-literal::
-
- DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-Imports
--------
-
-
-
-.. code:: ipython3
-
- # Fetch `notebook_utils` module
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
-
- open("notebook_utils.py", "w").write(r.text)
-
- from notebook_utils import download_file
-
-.. code:: ipython3
-
- import operator
- import time
- from urllib import parse
- from pathlib import Path
-
- import numpy as np
- import openvino as ov
-
-
- download_file(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/question-answering/html_reader.py",
- filename="html_reader.py",
- )
- import html_reader as reader
-
- download_file(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/question-answering/tokens_bert.py",
- filename="tokens_bert.py",
- )
- import tokens_bert as tokens
-
-
-
-.. parsed-literal::
-
- html_reader.py: 0%| | 0.00/632 [00:00`__.
-
-.. code:: ipython3
-
- # A generator of a sequence of inputs.
- def prepare_input(question_tokens, context_tokens):
- # A length of question in tokens.
- question_len = len(question_tokens)
- # The context part size.
- context_len = input_size - question_len - 3
-
- if context_len < 16:
- raise RuntimeError("Question is too long in comparison to input size. No space for context")
-
- # Take parts of the context with overlapping by 0.5.
- for start in range(0, max(1, len(context_tokens) - context_len), context_len // 2):
- # A part of the context.
- part_context_tokens = context_tokens[start : start + context_len]
- # The input: a question and the context separated by special tokens.
- input_ids = [cls_token] + question_tokens + [sep_token] + part_context_tokens + [sep_token]
- # 1 for any index if there is no padding token, 0 otherwise.
- attention_mask = [1] * len(input_ids)
- # 0 for question tokens, 1 for context part.
- token_type_ids = [0] * (question_len + 2) + [1] * (len(part_context_tokens) + 1)
-
- # Add padding at the end.
- (input_ids, attention_mask, token_type_ids), pad_number = pad(
- input_ids=input_ids,
- attention_mask=attention_mask,
- token_type_ids=token_type_ids,
- )
-
- # Create an input to feed the model.
- input_dict = {
- "input_ids": np.array([input_ids], dtype=np.int32),
- "attention_mask": np.array([attention_mask], dtype=np.int32),
- "token_type_ids": np.array([token_type_ids], dtype=np.int32),
- }
-
- # Some models require additional position_ids.
- if "position_ids" in [i_key.any_name for i_key in input_keys]:
- position_ids = np.arange(len(input_ids))
- input_dict["position_ids"] = np.array([position_ids], dtype=np.int32)
-
- yield input_dict, pad_number, start
-
-
- # A function to add padding.
- def pad(input_ids, attention_mask, token_type_ids):
- # How many padding tokens.
- diff_input_size = input_size - len(input_ids)
-
- if diff_input_size > 0:
- # Add padding to all the inputs.
- input_ids = input_ids + [pad_token] * diff_input_size
- attention_mask = attention_mask + [0] * diff_input_size
- token_type_ids = token_type_ids + [0] * diff_input_size
-
- return (input_ids, attention_mask, token_type_ids), diff_input_size
-
-Postprocessing
-~~~~~~~~~~~~~~
-
-
-
-The results from the network are raw (logits). Use the softmax function
-to get the probability distribution. Then, find the best answer in the
-current part of the context (the highest score) and return the score and
-the context range for the answer.
-
-.. code:: ipython3
-
- # Based on https://github.com/openvinotoolkit/open_model_zoo/blob/bf03f505a650bafe8da03d2747a8b55c5cb2ef16/demos/common/python/openvino/model_zoo/model_api/models/bert.py#L163
- def postprocess(
- output_start,
- output_end,
- question_tokens,
- context_tokens_start_end,
- padding,
- start_idx,
- ):
- def get_score(logits):
- out = np.exp(logits)
- return out / out.sum(axis=-1)
-
- # Get start-end scores for the context.
- score_start = get_score(output_start)
- score_end = get_score(output_end)
-
- # An index of the first context token in a tensor.
- context_start_idx = len(question_tokens) + 2
- # An index of the last+1 context token in a tensor.
- context_end_idx = input_size - padding - 1
-
- # Find product of all start-end combinations to find the best one.
- max_score, max_start, max_end = find_best_answer_window(
- start_score=score_start,
- end_score=score_end,
- context_start_idx=context_start_idx,
- context_end_idx=context_end_idx,
- )
-
- # Convert to context text start-end index.
- max_start = context_tokens_start_end[max_start + start_idx][0]
- max_end = context_tokens_start_end[max_end + start_idx][1]
-
- return max_score, max_start, max_end
-
-
- # Based on https://github.com/openvinotoolkit/open_model_zoo/blob/bf03f505a650bafe8da03d2747a8b55c5cb2ef16/demos/common/python/openvino/model_zoo/model_api/models/bert.py#L188
- def find_best_answer_window(start_score, end_score, context_start_idx, context_end_idx):
- context_len = context_end_idx - context_start_idx
- score_mat = np.matmul(
- start_score[context_start_idx:context_end_idx].reshape((context_len, 1)),
- end_score[context_start_idx:context_end_idx].reshape((1, context_len)),
- )
- # Reset candidates with end before start.
- score_mat = np.triu(score_mat)
- # Reset long candidates (>16 words).
- score_mat = np.tril(score_mat, 16)
- # Find the best start-end pair.
- max_s, max_e = divmod(score_mat.flatten().argmax(), score_mat.shape[1])
- max_score = score_mat[max_s, max_e]
-
- return max_score, max_s, max_e
-
-First, create a list of tokens from the context and the question. Then,
-find the best answer by trying different parts of the context. The best
-answer should come with the highest score.
-
-.. code:: ipython3
-
- def get_best_answer(question, context):
- # Convert the context string to tokens.
- context_tokens, context_tokens_start_end = tokens.text_to_tokens(text=context.lower(), vocab=vocab)
- # Convert the question string to tokens.
- question_tokens, _ = tokens.text_to_tokens(text=question.lower(), vocab=vocab)
-
- results = []
- # Iterate through different parts of the context.
- for network_input, padding, start_idx in prepare_input(question_tokens=question_tokens, context_tokens=context_tokens):
- # Get output layers.
- output_start_key = compiled_model.output("output_s")
- output_end_key = compiled_model.output("output_e")
-
- # OpenVINO inference.
- result = compiled_model(network_input)
- # Postprocess the result, getting the score and context range for the answer.
- score_start_end = postprocess(
- output_start=result[output_start_key][0],
- output_end=result[output_end_key][0],
- question_tokens=question_tokens,
- context_tokens_start_end=context_tokens_start_end,
- padding=padding,
- start_idx=start_idx,
- )
- results.append(score_start_end)
-
- # Find the highest score.
- answer = max(results, key=operator.itemgetter(0))
- # Return the part of the context, which is already an answer.
- return context[answer[1] : answer[2]], answer[0]
-
-Main Processing Function
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Run question answering on a specific knowledge base (websites) and
-iterate through the questions.
-
-.. code:: ipython3
-
- def run_question_answering(sources, example_question=None):
- print(f"Context: {sources}", flush=True)
- context = load_context(sources)
-
- if len(context) == 0:
- print("Error: Empty context or outside paragraphs")
- return
-
- if example_question is not None:
- start_time = time.perf_counter()
- answer, score = get_best_answer(question=example_question, context=context)
- end_time = time.perf_counter()
-
- print(f"Question: {example_question}")
- print(f"Answer: {answer}")
- print(f"Score: {score:.2f}")
- print(f"Time: {end_time - start_time:.2f}s")
- else:
- while True:
- question = input()
- # if no question - break
- if question == "":
- break
-
- # measure processing time
- start_time = time.perf_counter()
- answer, score = get_best_answer(question=question, context=context)
- end_time = time.perf_counter()
-
- print(f"Question: {question}")
- print(f"Answer: {answer}")
- print(f"Score: {score:.2f}")
- print(f"Time: {end_time - start_time:.2f}s")
-
-Run
----
-
-
-
-Run on local paragraphs
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-Change sources to your own to answer your questions. You can use as many
-sources as you want. Usually, you need to wait a few seconds for the
-answer, but the longer the context, the longer the waiting time. The
-model is very limited and sensitive for the input. The answer can depend
-on whether there is a question mark at the end. The model will try to
-answer any of your questions even if there is no good answer in the
-context. Therefore, in such cases, you can see random results.
-
-Sample source: a paragraph from `Computational complexity
-theory `__
-
-Sample questions:
-
-- What is the term for a task that generally lends itself to being
- solved by a computer?
-- By what main attribute are computational problems classified
- utilizing computational complexity theory?
-- What branch of theoretical computer science deals with broadly
- classifying computational problems by difficulty and class of
- relationship?
-
-If you want to stop the processing just put an empty string.
-
-**First, run the code below. If you want to run it in interactive mode
-set ``example_question`` as ``None``, run the code, and then put your
-questions in the box.**
-
-.. code:: ipython3
-
- sources = [
- "Computational complexity theory is a branch of the theory of computation in theoretical computer "
- "science that focuses on classifying computational problems according to their inherent difficulty, "
- "and relating those classes to each other. A computational problem is understood to be a task that "
- "is in principle amenable to being solved by a computer, which is equivalent to stating that the "
- "problem may be solved by mechanical application of mathematical steps, such as an algorithm."
- ]
-
- question = "What is the term for a task that generally lends itself to being solved by a computer?"
-
- run_question_answering(sources, example_question=question)
-
-
-.. parsed-literal::
-
- Context: ['Computational complexity theory is a branch of the theory of computation in theoretical computer science that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other. A computational problem is understood to be a task that is in principle amenable to being solved by a computer, which is equivalent to stating that the problem may be solved by mechanical application of mathematical steps, such as an algorithm.']
-
-
-.. parsed-literal::
-
- Question: What is the term for a task that generally lends itself to being solved by a computer?
- Answer: A computational problem
- Score: 0.53
- Time: 0.03s
-
-
-Run on websites
-~~~~~~~~~~~~~~~
-
-
-
-You can also provide URLs. Note that the context (a knowledge base) is
-built from paragraphs on websites. If some information is outside the
-paragraphs, the algorithm will not be able to find it.
-
-Sample source: `OpenVINO
-wiki `__
-
-Sample questions:
-
-- What does OpenVINO mean?
-- What is the license for OpenVINO?
-- Where can you deploy OpenVINO code?
-
-If you want to stop the processing just put an empty string.
-
-**First, run the code below. If you want to run it in interactive mode
-set ``example_question`` as ``None``, run the code, and then put your
-questions in the box.**
-
-.. code:: ipython3
-
- sources = ["https://en.wikipedia.org/wiki/OpenVINO"]
-
- question = "What does OpenVINO mean?"
-
- run_question_answering(sources, example_question=question)
-
-
-.. parsed-literal::
-
- Context: ['https://en.wikipedia.org/wiki/OpenVINO']
-
-
-.. parsed-literal::
-
- Question: What does OpenVINO mean?
- Answer: an open-source software toolkit for optimizing and deploying deep learning models
- Score: 0.09
- Time: 0.03s
-
diff --git a/docs/notebooks/rmbg-background-removal-with-output.rst b/docs/notebooks/rmbg-background-removal-with-output.rst
index f71222b4d0d..9b817de2de3 100644
--- a/docs/notebooks/rmbg-background-removal-with-output.rst
+++ b/docs/notebooks/rmbg-background-removal-with-output.rst
@@ -48,18 +48,9 @@ install required dependencies
%pip install -q torch torchvision pillow huggingface_hub "openvino>=2024.0.0" matplotlib "gradio>=4.15" "transformers>=4.39.1" tqdm --extra-index-url https://download.pytorch.org/whl/cpu
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -78,6 +69,19 @@ Download model code from HuggingFace hub
if not Path(file_for_downloading).exists():
hf_hub_download(repo_id=repo_id, filename=file_for_downloading, local_dir=".")
+
+
+.. parsed-literal::
+
+ utilities.py: 0%| | 0.00/980 [00:00=4.13" "openvino>=2023.1.0" "nncf>=2.7.0" "torch>=2.1" "torchvision>=0.16" Pillow opencv-python tqdm --extra-index-url https://download.pytorch.org/whl/cpu
-
+
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
@@ -172,18 +172,18 @@ model type below to a SAM model checkpoint, then load the model using
# Fetch `notebook_utils` module
import requests
-
+
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
-
+
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
-
+
checkpoint = "sam_vit_b_01ec64.pth"
model_url = "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth"
model_type = "vit_b"
-
+
download_file(model_url)
@@ -203,7 +203,7 @@ model type below to a SAM model checkpoint, then load the model using
.. code:: ipython3
from segment_anything import sam_model_registry
-
+
sam = sam_model_registry[model_type](checkpoint=checkpoint)
As we already discussed, Image Encoder part can be used once per image,
@@ -228,15 +228,15 @@ embeddings, tensor with shape ``1x256x64x64``
from pathlib import Path
import torch
import openvino as ov
-
+
core = ov.Core()
-
+
ov_encoder_path = Path("sam_image_encoder.xml")
if not ov_encoder_path.exists():
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=torch.jit.TracerWarning)
warnings.filterwarnings("ignore", category=UserWarning)
-
+
ov_encoder_model = ov.convert_model(
sam.image_encoder,
example_input=torch.zeros(1, 3, 1024, 1024),
@@ -249,14 +249,14 @@ embeddings, tensor with shape ``1x256x64x64``
.. code:: ipython3
import ipywidgets as widgets
-
+
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="AUTO",
description="Device:",
disabled=False,
)
-
+
device
@@ -310,8 +310,8 @@ Model outputs:
.. code:: ipython3
from typing import Tuple
-
-
+
+
class SamExportableModel(torch.nn.Module):
def __init__(
self,
@@ -328,25 +328,25 @@ Model outputs:
self.use_stability_score = use_stability_score
self.stability_score_offset = 1.0
self.return_extra_metrics = return_extra_metrics
-
+
def _embed_points(self, point_coords: torch.Tensor, point_labels: torch.Tensor) -> torch.Tensor:
point_coords = point_coords + 0.5
point_coords = point_coords / self.img_size
point_embedding = self.model.prompt_encoder.pe_layer._pe_encoding(point_coords)
point_labels = point_labels.unsqueeze(-1).expand_as(point_embedding)
-
+
point_embedding = point_embedding * (point_labels != -1).to(torch.float32)
point_embedding = point_embedding + self.model.prompt_encoder.not_a_point_embed.weight * (point_labels == -1).to(torch.float32)
-
+
for i in range(self.model.prompt_encoder.num_point_embeddings):
point_embedding = point_embedding + self.model.prompt_encoder.point_embeddings[i].weight * (point_labels == i).to(torch.float32)
-
+
return point_embedding
-
+
def t_embed_masks(self, input_mask: torch.Tensor) -> torch.Tensor:
mask_embedding = self.model.prompt_encoder.mask_downscaling(input_mask)
return mask_embedding
-
+
def mask_postprocessing(self, masks: torch.Tensor) -> torch.Tensor:
masks = torch.nn.functional.interpolate(
masks,
@@ -355,7 +355,7 @@ Model outputs:
align_corners=False,
)
return masks
-
+
def select_masks(self, masks: torch.Tensor, iou_preds: torch.Tensor, num_points: int) -> Tuple[torch.Tensor, torch.Tensor]:
# Determine if we should return the multiclick mask or not from the number of points.
# The reweighting is used to avoid control flow.
@@ -364,9 +364,9 @@ Model outputs:
best_idx = torch.argmax(score, dim=1)
masks = masks[torch.arange(masks.shape[0]), best_idx, :, :].unsqueeze(1)
iou_preds = iou_preds[torch.arange(masks.shape[0]), best_idx].unsqueeze(1)
-
+
return masks, iou_preds
-
+
@torch.no_grad()
def forward(
self,
@@ -382,30 +382,30 @@ Model outputs:
)
else:
dense_embedding = self._embed_masks(mask_input)
-
+
masks, scores = self.model.mask_decoder.predict_masks(
image_embeddings=image_embeddings,
image_pe=self.model.prompt_encoder.get_dense_pe(),
sparse_prompt_embeddings=sparse_embedding,
dense_prompt_embeddings=dense_embedding,
)
-
+
if self.use_stability_score:
scores = calculate_stability_score(masks, self.model.mask_threshold, self.stability_score_offset)
-
+
if self.return_single_mask:
masks, scores = self.select_masks(masks, scores, point_coords.shape[1])
-
+
upscaled_masks = self.mask_postprocessing(masks)
-
+
if self.return_extra_metrics:
stability_scores = calculate_stability_score(upscaled_masks, self.model.mask_threshold, self.stability_score_offset)
areas = (upscaled_masks > self.model.mask_threshold).sum(-1).sum(-1)
return upscaled_masks, scores, stability_scores, areas, masks
-
+
return upscaled_masks, scores
-
-
+
+
ov_model_path = Path("sam_mask_predictor.xml")
if not ov_model_path.exists():
exportable_model = SamExportableModel(sam, return_single_mask=True)
@@ -456,7 +456,7 @@ Example Image
import numpy as np
import cv2
import matplotlib.pyplot as plt
-
+
download_file("https://raw.githubusercontent.com/facebookresearch/segment-anything/main/notebooks/images/truck.jpg")
image = cv2.imread("truck.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
@@ -502,25 +502,25 @@ These steps are applicable to all available models
from copy import deepcopy
from typing import Tuple
from torchvision.transforms.functional import resize, to_pil_image
-
-
+
+
class ResizeLongestSide:
"""
Resizes images to longest side 'target_length', as well as provides
methods for resizing coordinates and boxes. Provides methods for
transforming numpy arrays.
"""
-
+
def __init__(self, target_length: int) -> None:
self.target_length = target_length
-
+
def apply_image(self, image: np.ndarray) -> np.ndarray:
"""
Expects a numpy array with shape HxWxC in uint8 format.
"""
target_size = self.get_preprocess_shape(image.shape[0], image.shape[1], self.target_length)
return np.array(resize(to_pil_image(image), target_size))
-
+
def apply_coords(self, coords: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray:
"""
Expects a numpy array of length 2 in the final dimension. Requires the
@@ -532,7 +532,7 @@ These steps are applicable to all available models
coords[..., 0] = coords[..., 0] * (new_w / old_w)
coords[..., 1] = coords[..., 1] * (new_h / old_h)
return coords
-
+
def apply_boxes(self, boxes: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray:
"""
Expects a numpy array shape Bx4. Requires the original image size
@@ -540,7 +540,7 @@ These steps are applicable to all available models
"""
boxes = self.apply_coords(boxes.reshape(-1, 2, 2), original_size)
return boxes.reshape(-1, 4)
-
+
@staticmethod
def get_preprocess_shape(oldh: int, oldw: int, long_side_length: int) -> Tuple[int, int]:
"""
@@ -551,11 +551,11 @@ These steps are applicable to all available models
neww = int(neww + 0.5)
newh = int(newh + 0.5)
return (newh, neww)
-
-
+
+
resizer = ResizeLongestSide(1024)
-
-
+
+
def preprocess_image(image: np.ndarray):
resized_image = resizer.apply_image(image)
resized_image = (resized_image.astype(np.float32) - [123.675, 116.28, 103.53]) / [
@@ -564,15 +564,15 @@ These steps are applicable to all available models
57.375,
]
resized_image = np.expand_dims(np.transpose(resized_image, (2, 0, 1)).astype(np.float32), 0)
-
+
# Pad
h, w = resized_image.shape[-2:]
padh = 1024 - h
padw = 1024 - w
x = np.pad(resized_image, ((0, 0), (0, 0), (0, padh), (0, padw)))
return x
-
-
+
+
def postprocess_masks(masks: np.ndarray, orig_size):
size_before_pad = resizer.get_preprocess_shape(orig_size[0], orig_size[1], masks.shape[-1])
masks = masks[..., : int(size_before_pad[0]), : int(size_before_pad[1])]
@@ -586,8 +586,8 @@ These steps are applicable to all available models
h, w = mask.shape[-2:]
mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
ax.imshow(mask_image)
-
-
+
+
def show_points(coords, labels, ax, marker_size=375):
pos_points = coords[labels == 1]
neg_points = coords[labels == 0]
@@ -609,8 +609,8 @@ These steps are applicable to all available models
edgecolor="white",
linewidth=1.25,
)
-
-
+
+
def show_box(box, ax):
x0, y0 = box[0], box[1]
w, h = box[2] - box[0], box[3] - box[1]
@@ -630,7 +630,7 @@ reuse them.
preprocessed_image = preprocess_image(image)
encoding_results = ov_encoder(preprocessed_image)
-
+
image_embeddings = encoding_results[ov_encoder.output(0)]
Now, we can try to provide different prompts for mask generation
@@ -647,7 +647,7 @@ location on the image below.
input_point = np.array([[500, 375]])
input_label = np.array([1])
-
+
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
@@ -684,7 +684,7 @@ object).
.. code:: ipython3
results = ov_predictor(inputs)
-
+
masks = results[ov_predictor.output(0)]
masks = postprocess_masks(masks, image.shape[:-1])
masks = masks > 0.0
@@ -737,7 +737,7 @@ Transform the points as in the previous example.
coord = np.concatenate([input_point, np.array([[0.0, 0.0]])], axis=0)[None, :, :]
label = np.concatenate([input_label, np.array([-1])], axis=0)[None, :].astype(np.float32)
-
+
coord = resizer.apply_coords(coord, image.shape[:2]).astype(np.float32)
Package inputs, then predict and threshold the mask.
@@ -749,9 +749,9 @@ Package inputs, then predict and threshold the mask.
"point_coords": coord,
"point_labels": label,
}
-
+
results = ov_predictor(inputs)
-
+
masks = results[ov_predictor.output(0)]
masks = postprocess_masks(masks, image.shape[:-1])
masks = masks > 0.0
@@ -810,10 +810,10 @@ padding point since the input includes a box input.
box_coords = input_box.reshape(2, 2)
box_labels = np.array([2, 3])
-
+
coord = np.concatenate([input_point, box_coords], axis=0)[None, :, :]
label = np.concatenate([input_label, box_labels], axis=0)[None, :].astype(np.float32)
-
+
coord = resizer.apply_coords(coord, image.shape[:2]).astype(np.float32)
Package inputs, then predict and threshold the mask.
@@ -825,9 +825,9 @@ Package inputs, then predict and threshold the mask.
"point_coords": coord,
"point_labels": label,
}
-
+
results = ov_predictor(inputs)
-
+
masks = results[ov_predictor.output(0)]
masks = postprocess_masks(masks, image.shape[:-1])
masks = masks > 0.0
@@ -859,14 +859,14 @@ point.
.. code:: ipython3
import gradio as gr
-
-
+
+
class Segmenter:
def __init__(self, ov_encoder, ov_predictor):
self.encoder = ov_encoder
self.predictor = ov_predictor
self._img_embeddings = None
-
+
def set_image(self, img: np.ndarray):
if self._img_embeddings is not None:
del self._img_embeddings
@@ -875,7 +875,7 @@ point.
image_embeddings = encoding_results[ov_encoder.output(0)]
self._img_embeddings = image_embeddings
return img
-
+
def get_mask(self, points, img):
coord = np.array(points)
coord = np.concatenate([coord, np.array([[0, 0]])], axis=0)
@@ -889,29 +889,29 @@ point.
"point_coords": coord,
"point_labels": label,
}
-
+
results = self.predictor(inputs)
masks = results[ov_predictor.output(0)]
masks = postprocess_masks(masks, img.shape[:-1])
-
+
masks = masks > 0.0
mask = masks[0]
mask = np.transpose(mask, (1, 2, 0))
return mask
-
-
+
+
segmenter = Segmenter(ov_encoder, ov_predictor)
-
-
+
+
with gr.Blocks() as demo:
with gr.Row():
input_img = gr.Image(label="Input", type="numpy", height=480, width=480)
output_img = gr.Image(label="Selected Segment", type="numpy", height=480, width=480)
-
+
def on_image_change(img):
segmenter.set_image(img)
return img
-
+
def get_select_coords(img, evt: gr.SelectData):
pixels_in_queue = set()
h, w = img.shape[:2]
@@ -927,10 +927,10 @@ point.
out = cv2.addWeighted(out.astype(np.float32), 0.7, mask_image.astype(np.float32), 0.3, 0.0)
out = out.astype(np.uint8)
return out
-
+
input_img.select(get_select_coords, [input_img], output_img)
input_img.upload(on_image_change, [input_img], [input_img])
-
+
if __name__ == "__main__":
try:
demo.launch()
@@ -941,7 +941,7 @@ point.
.. parsed-literal::
Running on local URL: http://127.0.0.1:7860
-
+
To create a public link, set `share=True` in `launch()`.
@@ -1002,12 +1002,12 @@ postprocessing masks to remove small disconnected regions and holes.
stability_score_thresh,
) -> MaskData:
orig_h, orig_w = orig_size
-
+
# Run model on this batch
transformed_points = resizer.apply_coords(points, im_size)
in_points = transformed_points
in_labels = np.ones(in_points.shape[0], dtype=int)
-
+
inputs = {
"image_embeddings": image_embedding,
"point_coords": in_points[:, None, :],
@@ -1017,7 +1017,7 @@ postprocessing masks to remove small disconnected regions and holes.
masks = postprocess_masks(res[ov_predictor.output(0)], orig_size)
masks = torch.from_numpy(masks)
iou_preds = torch.from_numpy(res[ov_predictor.output(1)])
-
+
# Serialize predictions and store in MaskData
data = MaskData(
masks=masks.flatten(0, 1),
@@ -1025,32 +1025,32 @@ postprocessing masks to remove small disconnected regions and holes.
points=torch.as_tensor(points.repeat(masks.shape[1], axis=0)),
)
del masks
-
+
# Filter by predicted IoU
if iou_thresh > 0.0:
keep_mask = data["iou_preds"] > iou_thresh
data.filter(keep_mask)
-
+
# Calculate stability score
data["stability_score"] = calculate_stability_score(data["masks"], mask_threshold, stability_score_offset)
if stability_score_thresh > 0.0:
keep_mask = data["stability_score"] >= stability_score_thresh
data.filter(keep_mask)
-
+
# Threshold masks and calculate boxes
data["masks"] = data["masks"] > mask_threshold
data["boxes"] = batched_mask_to_box(data["masks"])
-
+
# Filter boxes that touch crop boundaries
keep_mask = ~is_box_near_crop_edge(data["boxes"], crop_box, [0, 0, orig_w, orig_h])
if not torch.all(keep_mask):
data.filter(keep_mask)
-
+
# Compress to RLE
data["masks"] = uncrop_masks(data["masks"], crop_box, orig_h, orig_w)
data["rles"] = mask_to_rle_pytorch(data["masks"])
del data["masks"]
-
+
return data
.. code:: ipython3
@@ -1074,11 +1074,11 @@ postprocessing masks to remove small disconnected regions and holes.
cropped_im_size = cropped_im.shape[:2]
preprocessed_cropped_im = preprocess_image(cropped_im)
crop_embeddings = ov_encoder(preprocessed_cropped_im)[ov_encoder.output(0)]
-
+
# Get points for this crop
points_scale = np.array(cropped_im_size)[None, ::-1]
points_for_image = point_grids[crop_layer_idx] * points_scale
-
+
# Generate masks for this crop in batches
data = MaskData()
for (points,) in batch_iterator(points_per_batch, points_for_image):
@@ -1095,7 +1095,7 @@ postprocessing masks to remove small disconnected regions and holes.
)
data.cat(batch_data)
del batch_data
-
+
# Remove duplicates within this crop.
keep_by_nms = batched_nms(
data["boxes"].float(),
@@ -1104,12 +1104,12 @@ postprocessing masks to remove small disconnected regions and holes.
iou_threshold=box_nms_thresh,
)
data.filter(keep_by_nms)
-
+
# Return to the original image frame
data["boxes"] = uncrop_boxes_xyxy(data["boxes"], crop_box)
data["points"] = uncrop_points(data["points"], crop_box)
data["crop_boxes"] = torch.tensor([crop_box for _ in range(len(data["rles"]))])
-
+
return data
.. code:: ipython3
@@ -1117,13 +1117,13 @@ postprocessing masks to remove small disconnected regions and holes.
def generate_masks(image: np.ndarray, point_grids, crop_n_layers, crop_overlap_ratio, crop_nms_thresh) -> MaskData:
orig_size = image.shape[:2]
crop_boxes, layer_idxs = generate_crop_boxes(orig_size, crop_n_layers, crop_overlap_ratio)
-
+
# Iterate over image crops
data = MaskData()
for crop_box, layer_idx in zip(crop_boxes, layer_idxs):
crop_data = process_crop(image, point_grids, crop_box, layer_idx, orig_size)
data.cat(crop_data)
-
+
# Remove duplicate masks between crops
if len(crop_boxes) > 1:
# Prefer masks from smaller crops
@@ -1136,7 +1136,7 @@ postprocessing masks to remove small disconnected regions and holes.
iou_threshold=crop_nms_thresh,
)
data.filter(keep_by_nms)
-
+
data.to_numpy()
return data
@@ -1146,30 +1146,30 @@ postprocessing masks to remove small disconnected regions and holes.
"""
Removes small disconnected regions and holes in masks, then reruns
box NMS to remove any new duplicates.
-
+
Edits mask_data in place.
-
+
Requires open-cv as a dependency.
"""
if len(mask_data["rles"]) == 0:
return mask_data
-
+
# Filter small disconnected regions and holes
new_masks = []
scores = []
for rle in mask_data["rles"]:
mask = rle_to_mask(rle)
-
+
mask, changed = remove_small_regions(mask, min_area, mode="holes")
unchanged = not changed
mask, changed = remove_small_regions(mask, min_area, mode="islands")
unchanged = unchanged and not changed
-
+
new_masks.append(torch.as_tensor(mask).unsqueeze(0))
# Give score=0 to changed masks and score=1 to unchanged masks
# so NMS will prefer ones that didn't need postprocessing
scores.append(float(unchanged))
-
+
# Recalculate boxes and remove any new duplicates
masks = torch.cat(new_masks, dim=0)
boxes = batched_mask_to_box(masks)
@@ -1179,7 +1179,7 @@ postprocessing masks to remove small disconnected regions and holes.
torch.zeros(len(boxes)), # categories
iou_threshold=nms_thresh,
)
-
+
# Only recalculate RLEs for masks that have changed
for i_mask in keep_by_nms:
if scores[i_mask] == 0.0:
@@ -1188,7 +1188,7 @@ postprocessing masks to remove small disconnected regions and holes.
# update res directly
mask_data["boxes"][i_mask] = boxes[i_mask]
mask_data.filter(keep_by_nms)
-
+
return mask_data
There are several tunable parameters in automatic mask generation that
@@ -1211,10 +1211,10 @@ smaller objects, and post-processing can remove stray pixels and holes
) -> List[Dict[str, Any]]:
"""
Generates masks for the given image.
-
+
Arguments:
image (np.ndarray): The image to generate masks for, in HWC uint8 format.
-
+
Returns:
list(dict(str, any)): A list over records for masks. Each record is
a dict containing the following keys:
@@ -1238,7 +1238,7 @@ smaller objects, and post-processing can remove stray pixels and holes
crop_n_points_downscale_factor,
)
mask_data = generate_masks(image, point_grids, crop_n_layers, crop_overlap_ratio, crop_nms_thresh)
-
+
# Filter small disconnected regions and holes in masks
if min_mask_region_area > 0:
mask_data = postprocess_small_regions(
@@ -1246,9 +1246,9 @@ smaller objects, and post-processing can remove stray pixels and holes
min_mask_region_area,
max(box_nms_thresh, crop_nms_thresh),
)
-
+
mask_data["segmentations"] = [rle_to_mask(rle) for rle in mask_data["rles"]]
-
+
# Write mask records
curr_anns = []
for idx in range(len(mask_data["segmentations"])):
@@ -1262,7 +1262,7 @@ smaller objects, and post-processing can remove stray pixels and holes
"crop_box": box_xyxy_to_xywh(mask_data["crop_boxes"][idx]).tolist(),
}
curr_anns.append(ann)
-
+
return curr_anns
.. code:: ipython3
@@ -1297,8 +1297,8 @@ is a dictionary containing various data about the mask. These keys are:
.. code:: ipython3
from tqdm.notebook import tqdm
-
-
+
+
def draw_anns(image, anns):
if len(anns) == 0:
return
@@ -1313,10 +1313,10 @@ is a dictionary containing various data about the mask. These keys are:
.. code:: ipython3
import PIL
-
+
out = draw_anns(image, prediction)
cv2.imwrite("result.png", out[:, :, ::-1])
-
+
PIL.Image.open("result.png")
@@ -1364,12 +1364,12 @@ the label files.
.. code:: ipython3
from zipfile import ZipFile
-
+
DATA_URL = "https://ultralytics.com/assets/coco128.zip"
OUT_DIR = Path(".")
-
+
download_file(DATA_URL, directory=OUT_DIR, show_progress=True)
-
+
if not (OUT_DIR / "coco128/images/train2017").exists():
with ZipFile("coco128.zip", "r") as zip_ref:
zip_ref.extractall(OUT_DIR)
@@ -1387,22 +1387,22 @@ calibration dataset. For PyTorch, we can pass an instance of the
.. code:: ipython3
import torch.utils.data as data
-
-
+
+
class COCOLoader(data.Dataset):
def __init__(self, images_path):
self.images = list(Path(images_path).iterdir())
-
+
def __getitem__(self, index):
image_path = self.images[index]
image = cv2.imread(str(image_path))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
return image
-
+
def __len__(self):
return len(self.images)
-
-
+
+
coco_dataset = COCOLoader(OUT_DIR / "coco128/images/train2017")
calibration_loader = torch.utils.data.DataLoader(coco_dataset)
@@ -1412,8 +1412,8 @@ dataset and returns data that can be passed to the model for inference.
.. code:: ipython3
import nncf
-
-
+
+
def transform_fn(image_data):
"""
Quantization transform function. Extracts and preprocess input data from dataloader item for quantization.
@@ -1425,8 +1425,8 @@ dataset and returns data that can be passed to the model for inference.
image = image_data.numpy()
processed_image = preprocess_image(np.squeeze(image))
return processed_image
-
-
+
+
calibration_dataset = nncf.Dataset(calibration_loader, transform_fn)
@@ -1498,8 +1498,6 @@ activations.
-
-
.. code:: ipython3
@@ -1520,12 +1518,12 @@ We can reuse the previous code to validate the output of ``INT8`` model.
ov_encoder_int8 = core.compile_model(ov_encoder_model_int8, device.value)
encoding_results = ov_encoder_int8(preprocessed_image)
image_embeddings = encoding_results[ov_encoder_int8.output(0)]
-
+
input_point = np.array([[500, 375]])
input_label = np.array([1])
coord = np.concatenate([input_point, np.array([[0.0, 0.0]])], axis=0)[None, :, :]
label = np.concatenate([input_label, np.array([-1])], axis=0)[None, :].astype(np.float32)
-
+
coord = resizer.apply_coords(coord, image.shape[:2]).astype(np.float32)
inputs = {
"image_embeddings": image_embeddings,
@@ -1533,7 +1531,7 @@ We can reuse the previous code to validate the output of ``INT8`` model.
"point_labels": label,
}
results = ov_predictor(inputs)
-
+
masks = results[ov_predictor.output(0)]
masks = postprocess_masks(masks, image.shape[:-1])
masks = masks > 0.0
@@ -1595,12 +1593,12 @@ models.
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.1.0-12050-e33de350633
- [ INFO ]
+ [ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2023.1.0-12050-e33de350633
- [ INFO ]
- [ INFO ]
+ [ INFO ]
+ [ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
@@ -1648,7 +1646,7 @@ models.
[ INFO ] LOADED_FROM_CACHE: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
- [ INFO ] Fill input 'x' with random values
+ [ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 3347.39 ms
@@ -1678,12 +1676,12 @@ models.
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.1.0-12050-e33de350633
- [ INFO ]
+ [ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
[ INFO ] Build ................................. 2023.1.0-12050-e33de350633
- [ INFO ]
- [ INFO ]
+ [ INFO ]
+ [ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
@@ -1731,7 +1729,7 @@ models.
[ INFO ] LOADED_FROM_CACHE: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!
- [ INFO ] Fill input 'x' with random values
+ [ INFO ] Fill input 'x' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 1951.78 ms
diff --git a/docs/notebooks/segmenter-semantic-segmentation-with-output.rst b/docs/notebooks/segmenter-semantic-segmentation-with-output.rst
deleted file mode 100644
index 36f20d1123d..00000000000
--- a/docs/notebooks/segmenter-semantic-segmentation-with-output.rst
+++ /dev/null
@@ -1,975 +0,0 @@
-Semantic Segmentation with OpenVINO™ using Segmenter
-====================================================
-
-Semantic segmentation is a difficult computer vision problem with many
-applications such as autonomous driving, robotics, augmented reality,
-and many others. Its goal is to assign labels to each pixel according to
-the object it belongs to, creating so-called segmentation masks. To
-properly assign this label, the model needs to consider the local as
-well as global context of the image. This is where transformers offer
-their advantage as they work well in capturing global context.
-
-Segmenter is based on Vision Transformer working as an encoder, and Mask
-Transformer working as a decoder. With this configuration, it achieves
-good results on different datasets such as ADE20K, Pascal Context, and
-Cityscapes. It works as shown in the diagram below, by taking the image,
-splitting it into patches, and then encoding these patches. Mask
-transformer combines encoded patches with class masks and decodes them
-into a segmentation map as the output, where each pixel has a label
-assigned to it.
-
-|Segmenter diagram| > Credits for this image go to `original authors of
-Segmenter `__.
-
-More about the model and its details can be found in the following
-paper: `Segmenter: Transformer for Semantic
-Segmentation `__ or in the
-`repository `__.
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Get and prepare PyTorch model <#get-and-prepare-pytorch-model>`__
-
- - `Prerequisites <#prerequisites>`__
- - `Loading PyTorch model <#loading-pytorch-model>`__
-
-- `Preparing preprocessing and visualization
- functions <#preparing-preprocessing-and-visualization-functions>`__
-
- - `Preprocessing <#preprocessing>`__
- - `Visualization <#visualization>`__
-
-- `Validation of inference of original
- model <#validation-of-inference-of-original-model>`__
-- `Convert PyTorch model to OpenVINO Intermediate Representation
- (IR) <#convert-pytorch-model-to-openvino-intermediate-representation-ir>`__
-- `Verify converted model
- inference <#verify-converted-model-inference>`__
-
- - `Select inference device <#select-inference-device>`__
-
-- `Benchmarking performance of converted
- model <#benchmarking-performance-of-converted-model>`__
-
-.. |Segmenter diagram| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/93932510/f57979e7-fd3b-449f-bf01-afe0f965abbc
-
-To demonstrate how to convert and use Segmenter in OpenVINO, this
-notebook consists of the following steps:
-
-- Preparing PyTorch Segmenter model
-- Preparing preprocessing and visualization functions
-- Validating inference of original model
-- Converting PyTorch model to OpenVINO IR
-- Validating inference of the converted model
-- Benchmark performance of the converted model
-
-Get and prepare PyTorch model
------------------------------
-
-
-
-The first thing we’ll need to do is clone
-`repository `__ containing model
-and helper functions. We will use Tiny model with mask transformer, that
-is ``Seg-T-Mask/16``. There are also better, but much larger models
-available in the linked repo. This model is pre-trained on
-`ADE20K `__
-dataset used for segmentation.
-
-The code from the repository already contains functions that create
-model and load weights, but we will need to download config and trained
-weights (checkpoint) file and add some additional helper functions.
-
-Prerequisites
-~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- import sys
- from pathlib import Path
-
- # clone Segmenter repo
- if not Path("segmenter").exists():
- !git clone https://github.com/rstrudel/segmenter
- else:
- print("Segmenter repo already cloned")
-
- # include path to Segmenter repo to use its functions
- sys.path.append("./segmenter")
-
-
-.. parsed-literal::
-
- Cloning into 'segmenter'...
-
-
-.. parsed-literal::
-
- remote: Enumerating objects: 268, done.[K
- Receiving objects: 0% (1/268)
-Receiving objects: 1% (3/268)
-Receiving objects: 2% (6/268)
-Receiving objects: 3% (9/268)
-Receiving objects: 4% (11/268)
-Receiving objects: 5% (14/268)
-Receiving objects: 6% (17/268)
-Receiving objects: 7% (19/268)
-Receiving objects: 8% (22/268)
-Receiving objects: 9% (25/268)
-Receiving objects: 10% (27/268)
-Receiving objects: 11% (30/268)
-Receiving objects: 12% (33/268)
-Receiving objects: 13% (35/268)
-Receiving objects: 14% (38/268)
-Receiving objects: 15% (41/268)
-Receiving objects: 16% (43/268)
-Receiving objects: 17% (46/268)
-Receiving objects: 18% (49/268)
-Receiving objects: 19% (51/268)
-Receiving objects: 20% (54/268)
-Receiving objects: 21% (57/268)
-Receiving objects: 22% (59/268)
-
-.. parsed-literal::
-
- Receiving objects: 23% (62/268)
-
-.. parsed-literal::
-
- Receiving objects: 24% (65/268)
-
-.. parsed-literal::
-
- Receiving objects: 25% (67/268), 5.86 MiB | 11.28 MiB/s
-
-.. parsed-literal::
-
- Receiving objects: 26% (70/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 27% (73/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 28% (76/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 29% (78/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 30% (81/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 31% (84/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 32% (86/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 33% (89/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 34% (92/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 35% (94/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 36% (97/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 37% (100/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 38% (102/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 39% (105/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 40% (108/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 41% (110/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 42% (113/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 43% (116/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 44% (118/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 45% (121/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 46% (124/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 47% (126/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 48% (129/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 49% (132/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 50% (134/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 51% (137/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 52% (140/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 53% (143/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 54% (145/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 55% (148/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 56% (151/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 57% (153/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 58% (156/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 59% (159/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 60% (161/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 61% (164/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 62% (167/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 63% (169/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 64% (172/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 65% (175/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 66% (177/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 67% (180/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 68% (183/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 69% (185/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 70% (188/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 71% (191/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 72% (193/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 73% (196/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 74% (199/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 75% (201/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 76% (204/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 77% (207/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 78% (210/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 79% (212/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 80% (215/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 81% (218/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 82% (220/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 83% (223/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 84% (226/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 85% (228/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 86% (231/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 87% (234/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 88% (236/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 89% (239/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 90% (242/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 91% (244/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 92% (247/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 93% (250/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 94% (252/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 95% (255/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 96% (258/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 97% (260/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 98% (263/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 99% (266/268), 5.86 MiB | 11.28 MiB/s
-
-.. parsed-literal::
-
- remote: Total 268 (delta 0), reused 0 (delta 0), pack-reused 268[K
- Receiving objects: 100% (268/268), 5.86 MiB | 11.28 MiB/s
-Receiving objects: 100% (268/268), 15.34 MiB | 21.23 MiB/s, done.
- Resolving deltas: 0% (0/117)
-Resolving deltas: 1% (2/117)
-Resolving deltas: 2% (3/117)
-Resolving deltas: 5% (6/117)
-Resolving deltas: 7% (9/117)
-Resolving deltas: 8% (10/117)
-Resolving deltas: 9% (11/117)
-Resolving deltas: 10% (12/117)
-Resolving deltas: 11% (13/117)
-Resolving deltas: 13% (16/117)
-Resolving deltas: 14% (17/117)
-Resolving deltas: 27% (32/117)
-Resolving deltas: 30% (36/117)
-Resolving deltas: 56% (66/117)
-Resolving deltas: 58% (69/117)
-Resolving deltas: 59% (70/117)
-Resolving deltas: 73% (86/117)
-Resolving deltas: 76% (90/117)
-Resolving deltas: 78% (92/117)
-Resolving deltas: 81% (95/117)
-Resolving deltas: 100% (117/117)
-Resolving deltas: 100% (117/117), done.
-
-
-.. code:: ipython3
-
- # Installing requirements
- %pip install -q "openvino>=2023.1.0" tqdm
- %pip install -r segmenter/requirements.txt
-
-
-.. parsed-literal::
-
- DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
- Requirement already satisfied: torch in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 1)) (2.2.2+cpu)
- Requirement already satisfied: click in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 2)) (8.1.7)
- Requirement already satisfied: numpy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 3)) (1.23.5)
- Requirement already satisfied: einops in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 4)) (0.3.2)
-
-
-.. parsed-literal::
-
- Collecting python-hostlist (from -r segmenter/requirements.txt (line 5))
- Using cached python_hostlist-1.23.0-py3-none-any.whl
- Requirement already satisfied: tqdm in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 6)) (4.66.2)
- Requirement already satisfied: requests in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 7)) (2.31.0)
- Requirement already satisfied: pyyaml in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from -r segmenter/requirements.txt (line 8)) (6.0.1)
-
-
-.. parsed-literal::
-
- Collecting timm==0.4.12 (from -r segmenter/requirements.txt (line 9))
- Using cached timm-0.4.12-py3-none-any.whl.metadata (30 kB)
-
-
-.. parsed-literal::
-
- Collecting mmcv==1.3.8 (from -r segmenter/requirements.txt (line 10))
- Using cached mmcv-1.3.8-py2.py3-none-any.whl
-
-
-.. parsed-literal::
-
- Collecting mmsegmentation==0.14.1 (from -r segmenter/requirements.txt (line 11))
- Using cached mmsegmentation-0.14.1-py3-none-any.whl.metadata (8.3 kB)
- Requirement already satisfied: torchvision in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from timm==0.4.12->-r segmenter/requirements.txt (line 9)) (0.17.2+cpu)
- Requirement already satisfied: addict in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (2.4.0)
- Requirement already satisfied: Pillow in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (10.3.0)
- Requirement already satisfied: yapf in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (0.40.2)
- Requirement already satisfied: matplotlib in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (3.7.0)
- Requirement already satisfied: prettytable in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (3.10.0)
- Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (3.13.4)
- Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (4.11.0)
- Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (1.12)
- Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (3.1)
- Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (3.1.3)
- Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch->-r segmenter/requirements.txt (line 1)) (2024.2.0)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->-r segmenter/requirements.txt (line 7)) (3.3.2)
- Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->-r segmenter/requirements.txt (line 7)) (3.7)
- Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->-r segmenter/requirements.txt (line 7)) (2.2.1)
- Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests->-r segmenter/requirements.txt (line 7)) (2024.2.2)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch->-r segmenter/requirements.txt (line 1)) (2.1.5)
- Requirement already satisfied: contourpy>=1.0.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (1.1.1)
- Requirement already satisfied: cycler>=0.10 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (0.12.1)
- Requirement already satisfied: fonttools>=4.22.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (4.51.0)
- Requirement already satisfied: kiwisolver>=1.0.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (1.4.5)
- Requirement already satisfied: packaging>=20.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (24.0)
- Requirement already satisfied: pyparsing>=2.3.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (3.1.2)
- Requirement already satisfied: python-dateutil>=2.7 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (2.9.0.post0)
- Requirement already satisfied: importlib-resources>=3.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (6.4.0)
- Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prettytable->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (0.2.13)
- Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch->-r segmenter/requirements.txt (line 1)) (1.3.0)
- Requirement already satisfied: importlib-metadata>=6.6.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from yapf->mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (7.1.0)
- Requirement already satisfied: platformdirs>=3.5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from yapf->mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (4.2.0)
- Requirement already satisfied: tomli>=2.0.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from yapf->mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (2.0.1)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: zipp>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from importlib-metadata>=6.6.0->yapf->mmcv==1.3.8->-r segmenter/requirements.txt (line 10)) (3.18.1)
- Requirement already satisfied: six>=1.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from python-dateutil>=2.7->matplotlib->mmsegmentation==0.14.1->-r segmenter/requirements.txt (line 11)) (1.16.0)
-
-
-.. parsed-literal::
-
- Using cached timm-0.4.12-py3-none-any.whl (376 kB)
- Using cached mmsegmentation-0.14.1-py3-none-any.whl (201 kB)
-
-
-.. parsed-literal::
-
- DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
- Installing collected packages: python-hostlist, mmsegmentation, mmcv, timm
-
-
-.. parsed-literal::
-
- Attempting uninstall: timm
- Found existing installation: timm 0.9.16
- Uninstalling timm-0.9.16:
- Successfully uninstalled timm-0.9.16
-
-
-.. parsed-literal::
-
- ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
- mobileclip 0.1.0 requires timm>=0.9.5, but you have timm 0.4.12 which is incompatible.
- mobileclip 0.1.0 requires torch==1.13.1, but you have torch 2.2.2+cpu which is incompatible.
- mobileclip 0.1.0 requires torchvision==0.14.1, but you have torchvision 0.17.2+cpu which is incompatible.
- Successfully installed mmcv-1.3.8 mmsegmentation-0.14.1 python-hostlist-1.23.0 timm-0.4.12
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-.. code:: ipython3
-
- import numpy as np
- import yaml
-
- # Fetch the notebook utils script from the openvino_notebooks repo
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
-
- open("notebook_utils.py", "w").write(r.text)
- from notebook_utils import download_file, load_image
-
-We’ll need ``timm``, ``mmsegmentation``, ``einops`` and ``mmcv``, to use
-functions from segmenter repo
-
-First, we will clone the Segmenter repo and then download weights and
-config for our model.
-
-.. code:: ipython3
-
- # download config and pretrained model weights
- # here we use tiny model, there are also better but larger models available in repository
- WEIGHTS_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/segmenter/checkpoints/ade20k/seg_tiny_mask/checkpoint.pth"
- CONFIG_LINK = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/segmenter/checkpoints/ade20k/seg_tiny_mask/variant.yml"
-
- MODEL_DIR = Path("model/")
- MODEL_DIR.mkdir(exist_ok=True)
-
- download_file(WEIGHTS_LINK, directory=MODEL_DIR, show_progress=True)
- download_file(CONFIG_LINK, directory=MODEL_DIR, show_progress=True)
-
- WEIGHT_PATH = MODEL_DIR / "checkpoint.pth"
- CONFIG_PATH = MODEL_DIR / "variant.yaml"
-
-
-
-.. parsed-literal::
-
- model/checkpoint.pth: 0%| | 0.00/26.4M [00:00`__
-class, initialized by a state dictionary containing model weights.
-Typical steps to get the model are therefore:
-
-1. Create an instance of the model class
-2. Load checkpoint state dict, which contains pre-trained model weights
-3. Turn the model to evaluation mode, to switch some operations to
- inference mode
-
-We will now use already provided helper functions from repository to
-initialize the model.
-
-.. code:: ipython3
-
- from segmenter.segm.model.factory import load_model
-
- pytorch_model, config = load_model(WEIGHT_PATH)
- # put model into eval mode, to set it for inference
- pytorch_model.eval()
- print("PyTorch model loaded and ready for inference.")
-
-
-.. parsed-literal::
-
- PyTorch model loaded and ready for inference.
-
-
-Load normalization settings from config file.
-
-.. code:: ipython3
-
- from segmenter.segm.data.utils import STATS
-
- # load normalization name, in our case "vit" since we are using transformer
- normalization_name = config["dataset_kwargs"]["normalization"]
- # load normalization params, mean and std from STATS
- normalization = STATS[normalization_name]
-
-Preparing preprocessing and visualization functions
----------------------------------------------------
-
-
-
-Now we will define utility functions for preprocessing and visualizing
-the results.
-
-Preprocessing
-~~~~~~~~~~~~~
-
-
-
-Inference input is tensor with shape ``[1, 3, H, W]`` in ``B, C, H, W``
-format, where:
-
-- ``B`` - batch size (in our case 1, as we are just adding 1 with
- unsqueeze)
-- ``C`` - image channels (in our case RGB - 3)
-- ``H`` - image height
-- ``W`` - image width
-
-Resizing to the correct scale and splitting to batches is done inside
-inference, so we don’t need to resize or split the image in
-preprocessing.
-
-Model expects images in RGB channels format, scaled to [0, 1] range and
-normalized with given mean and standard deviation provided in
-``config.yml``.
-
-.. code:: ipython3
-
- from PIL import Image
- import torch
- import torchvision.transforms.functional as F
-
-
- def preprocess(im: Image, normalization: dict) -> torch.Tensor:
- """
- Preprocess image: scale, normalize and unsqueeze
-
- :param im: input image
- :param normalization: dictionary containing normalization data from config file
- :return:
- im: processed (scaled and normalized) image
- """
- # change PIL image to tensor and scale to [0, 1]
- im = F.pil_to_tensor(im).float() / 255
- # normalize by given mean and standard deviation
- im = F.normalize(im, normalization["mean"], normalization["std"])
- # change dim from [C, H, W] to [1, C, H, W]
- im = im.unsqueeze(0)
-
- return im
-
-Visualization
-~~~~~~~~~~~~~
-
-
-
-Inference output contains labels assigned to each pixel, so the output
-in our case is ``[150, H, W]`` in ``CL, H, W`` format where:
-
-- ``CL`` - number of classes for labels (in our case 150)
-- ``H`` - image height
-- ``W`` - image width
-
-Since we want to visualize this output, we reduce dimensions to
-``[1, H, W]`` where we keep only class with the highest value as that is
-the predicted label. We then combine original image with colors
-corresponding to the inferred labels.
-
-.. code:: ipython3
-
- from segmenter.segm.data.utils import dataset_cat_description, seg_to_rgb
- from segmenter.segm.data.ade20k import ADE20K_CATS_PATH
-
-
- def apply_segmentation_mask(pil_im: Image, results: torch.Tensor) -> Image:
- """
- Combine segmentation masks with the image
-
- :param pil_im: original input image
- :param results: tensor containing segmentation masks for each pixel
- :return:
- pil_blend: image with colored segmentation masks overlay
- """
- cat_names, cat_colors = dataset_cat_description(ADE20K_CATS_PATH)
-
- # 3D array, where each pixel has values for all classes, take index of max as label
- seg_map = results.argmax(0, keepdim=True)
- # transform label id to colors
- seg_rgb = seg_to_rgb(seg_map, cat_colors)
- seg_rgb = (255 * seg_rgb.cpu().numpy()).astype(np.uint8)
- pil_seg = Image.fromarray(seg_rgb[0])
-
- # overlay segmentation mask over original image
- pil_blend = Image.blend(pil_im, pil_seg, 0.5).convert("RGB")
-
- return pil_blend
-
-Validation of inference of original model
------------------------------------------
-
-
-
-Now that we have everything ready, we can perform segmentation on
-example image ``coco_hollywood.jpg``.
-
-.. code:: ipython3
-
- from segmenter.segm.model.utils import inference
-
- # load image with PIL
- image = load_image("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_hollywood.jpg")
- # load_image reads the image in BGR format, [:,:,::-1] reshape transfroms it to RGB
- pil_image = Image.fromarray(image[:, :, ::-1])
-
- # preprocess image with normalization params loaded in previous steps
- image = preprocess(pil_image, normalization)
-
- # inference function needs some meta parameters, where we specify that we don't flip images in inference mode
- im_meta = dict(flip=False)
- # perform inference with function from repository
- original_results = inference(
- model=pytorch_model,
- ims=[image],
- ims_metas=[im_meta],
- ori_shape=image.shape[2:4],
- window_size=config["inference_kwargs"]["window_size"],
- window_stride=config["inference_kwargs"]["window_stride"],
- batch_size=2,
- )
-
-After inference is complete, we need to transform output to segmentation
-mask where each class has specified color, using helper functions from
-previous steps.
-
-.. code:: ipython3
-
- # combine segmentation mask with image
- blended_image = apply_segmentation_mask(pil_image, original_results)
-
- # show image with segmentation mask overlay
- blended_image
-
-
-
-
-.. image:: segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.png
-
-
-
-We can see that model segments the image into meaningful parts. Since we
-are using tiny variant of model, the result is not as good as it is with
-larger models, but it already shows nice segmentation performance.
-
-Convert PyTorch model to OpenVINO Intermediate Representation (IR)
-------------------------------------------------------------------
-
-
-
-Now that we’ve verified that the inference of PyTorch model works, we
-will convert it to OpenVINO IR format.
-
-To do this, we first get input dimensions from the model configuration
-file and create torch dummy input. Input dimensions are in our case
-``[2, 3, 512, 512]`` in ``B, C, H, W]`` format, where:
-
-- ``B`` - batch size
-- ``C`` - image channels (in our case RGB - 3)
-- ``H`` - model input image height
-- ``W`` - model input image width
-
-..
-
- Note that H and W are here fixed to 512, as this is required by the
- model. Resizing is done inside the inference function from the
- original repository.
-
-After that, we use ``ov.convert_model`` function from PyTorch to convert
-the model to OpenVINO model, which is ready to use in Python interface
-but can also be serialized to OpenVINO IR format for future execution
-using ``ov.save_model``. The process can generate some warnings, but
-they are not a problem.
-
-.. code:: ipython3
-
- import openvino as ov
-
- # get input sizes from config file
- batch_size = 2
- channels = 3
- image_size = config["dataset_kwargs"]["image_size"]
-
- # make dummy input with correct shapes obtained from config file
- dummy_input = torch.randn(batch_size, channels, image_size, image_size)
-
- model = ov.convert_model(
- pytorch_model,
- example_input=dummy_input,
- input=([batch_size, channels, image_size, image_size],),
- )
- # serialize model for saving IR
- ov.save_model(model, MODEL_DIR / "segmenter.xml")
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:69: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if H % patch_size > 0:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:71: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if W % patch_size > 0:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/vit.py:122: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if x.shape[1] != pos_embed.shape[1]:
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/decoder.py:100: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- masks = rearrange(masks, "b (h w) n -> b n h w", h=int(GS))
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:85: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if extra_h > 0:
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/notebooks/segmenter-semantic-segmentation/./segmenter/segm/model/utils.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
- if extra_w > 0:
-
-
-Verify converted model inference
---------------------------------
-
-
-
-To test that model was successfully converted, we can use same inference
-function from original repository, but we need to make custom class.
-
-``SegmenterOV`` class contains OpenVINO model, with all attributes and
-methods required by inference function. This way we don’t need to write
-any additional custom code required to process input.
-
-.. code:: ipython3
-
- class SegmenterOV:
- """
- Class containing OpenVINO model with all attributes required to work with inference function.
-
- :param model: compiled OpenVINO model
- :type model: CompiledModel
- :param output_blob: output blob used in inference
- :type output_blob: ConstOutput
- :param config: config file containing data about model and its requirements
- :type config: dict
- :param n_cls: number of classes to be predicted
- :type n_cls: int
- :param normalization:
- :type normalization: dict
-
- """
-
- def __init__(self, model_path: Path, device: str = "CPU"):
- """
- Constructor method.
- Initializes OpenVINO model and sets all required attributes
-
- :param model_path: path to model's .xml file, also containing variant.yml
- :param device: device string for selecting inference device
- """
- # init OpenVino core
- core = ov.Core()
- # read model
- model_xml = core.read_model(model_path)
- self.model = core.compile_model(model_xml, device)
- self.output_blob = self.model.output(0)
-
- # load model configs
- variant_path = Path(model_path).parent / "variant.yml"
- with open(variant_path, "r") as f:
- self.config = yaml.load(f, Loader=yaml.FullLoader)
-
- # load normalization specs from config
- normalization_name = self.config["dataset_kwargs"]["normalization"]
- self.normalization = STATS[normalization_name]
-
- # load number of classes from config
- self.n_cls = self.config["net_kwargs"]["n_cls"]
-
- def forward(self, data: torch.Tensor) -> torch.Tensor:
- """
- Perform inference on data and return the result in Tensor format
-
- :param data: input data to model
- :return: data inferred by model
- """
- return torch.from_numpy(self.model(data)[self.output_blob])
-
-Now that we have created ``SegmenterOV`` helper class, we can use it in
-inference function.
-
-Select inference device
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- core = ov.Core()
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- # load model into SegmenterOV class
- model = SegmenterOV(MODEL_DIR / "segmenter.xml", device.value)
-
-.. code:: ipython3
-
- # perform inference with same function as in case of PyTorch model from repository
- results = inference(
- model=model,
- ims=[image],
- ims_metas=[im_meta],
- ori_shape=image.shape[2:4],
- window_size=model.config["inference_kwargs"]["window_size"],
- window_stride=model.config["inference_kwargs"]["window_stride"],
- batch_size=2,
- )
-
-.. code:: ipython3
-
- # combine segmentation mask with image
- converted_blend = apply_segmentation_mask(pil_image, results)
-
- # show image with segmentation mask overlay
- converted_blend
-
-
-
-
-.. image:: segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.png
-
-
-
-As we can see, we get the same results as with original model.
-
-Benchmarking performance of converted model
--------------------------------------------
-
-
-
-Finally, use the OpenVINO `Benchmark
-Tool `__
-to measure the inference performance of the model.
-
- NOTE: For more accurate performance, it is recommended to run
- ``benchmark_app`` in a terminal/command prompt after closing other
- applications. Run ``benchmark_app -m model.xml -d CPU`` to benchmark
- async inference on CPU for one minute. Change ``CPU`` to ``GPU`` to
- benchmark on GPU. Run ``benchmark_app --help`` to see an overview of
- all command-line options.
-
-..
-
- Keep in mind that the authors of original paper used V100 GPU, which
- is significantly more powerful than the CPU used to obtain the
- following throughput. Therefore, FPS can’t be compared directly.
-
-.. code:: ipython3
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- # Inference FP32 model (OpenVINO IR)
- !benchmark_app -m ./model/segmenter.xml -d $device.value -api async
-
-
-.. parsed-literal::
-
- [Step 1/11] Parsing and validating input arguments
- [ INFO ] Parsing input parameters
- [Step 2/11] Loading OpenVINO Runtime
- [ WARNING ] Default duration 120 seconds is used for unknown device AUTO
- [ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ] Device info:
- [ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
- [ INFO ]
- [ INFO ]
- [Step 3/11] Setting device configuration
- [ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
- [Step 4/11] Reading model files
- [ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 23.64 ms
- [ INFO ] Original model I/O parameters:
- [ INFO ] Model inputs:
- [ INFO ] im (node: im) : f32 / [...] / [2,3,512,512]
- [ INFO ] Model outputs:
- [ INFO ] y (node: aten::upsample_bilinear2d/Interpolate) : f32 / [...] / [2,150,512,512]
- [Step 5/11] Resizing model to match image sizes and given batch
- [ INFO ] Model batch size: 2
- [Step 6/11] Configuring input of the model
- [ INFO ] Model inputs:
- [ INFO ] im (node: im) : u8 / [N,C,H,W] / [2,3,512,512]
- [ INFO ] Model outputs:
- [ INFO ] y (node: aten::upsample_bilinear2d/Interpolate) : f32 / [...] / [2,150,512,512]
- [Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 356.63 ms
- [Step 8/11] Querying optimal runtime parameters
- [ INFO ] Model:
- [ INFO ] NETWORK_NAME: Model0
- [ INFO ] EXECUTION_DEVICES: ['CPU']
- [ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
- [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
- [ INFO ] MULTI_DEVICE_PRIORITIES: CPU
- [ INFO ] CPU:
- [ INFO ] AFFINITY: Affinity.CORE
- [ INFO ] CPU_DENORMALS_OPTIMIZATION: False
- [ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
- [ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
- [ INFO ] ENABLE_CPU_PINNING: True
- [ INFO ] ENABLE_HYPER_THREADING: True
- [ INFO ] EXECUTION_DEVICES: ['CPU']
- [ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
- [ INFO ] INFERENCE_NUM_THREADS: 24
- [ INFO ] INFERENCE_PRECISION_HINT:
- [ INFO ] KV_CACHE_PRECISION:
- [ INFO ] LOG_LEVEL: Level.NO
- [ INFO ] NETWORK_NAME: Model0
- [ INFO ] NUM_STREAMS: 6
- [ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
- [ INFO ] PERFORMANCE_HINT: THROUGHPUT
- [ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
- [ INFO ] PERF_COUNT: NO
- [ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
- [ INFO ] MODEL_PRIORITY: Priority.MEDIUM
- [ INFO ] LOADED_FROM_CACHE: False
- [Step 9/11] Creating infer requests and preparing input tensors
- [ WARNING ] No input files were given for input 'im'!. This input will be filled with random values!
- [ INFO ] Fill input 'im' with random values
- [Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 120000 ms duration)
- [ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 202.08 ms
-
-
-.. parsed-literal::
-
- [Step 11/11] Dumping statistics report
- [ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 1698 iterations
- [ INFO ] Duration: 120587.19 ms
- [ INFO ] Latency:
- [ INFO ] Median: 425.03 ms
- [ INFO ] Average: 425.44 ms
- [ INFO ] Min: 356.57 ms
- [ INFO ] Max: 515.02 ms
- [ INFO ] Throughput: 28.16 FPS
-
diff --git a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.jpg b/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.jpg
deleted file mode 100644
index e9278ec011d..00000000000
--- a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4eca61d116f6159e80e197fde99ce61b4eed1c83620a98ef6e0e37f462c96af0
-size 72352
diff --git a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.png b/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.png
deleted file mode 100644
index 0e2ded779fc..00000000000
--- a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_21_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:22bc159f17d6094accafc2ca4549d909e74b7cba49a23c3d4091f0ca70ff43a6
-size 909669
diff --git a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.jpg b/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.jpg
deleted file mode 100644
index 66f1523fe3f..00000000000
--- a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.jpg
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7e45440c38464f20541ffc81d464dd9889fabd08058af09c8bbecac3a7cd87c2
-size 72372
diff --git a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.png b/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.png
deleted file mode 100644
index 056bb9f567b..00000000000
--- a/docs/notebooks/segmenter-semantic-segmentation-with-output_files/segmenter-semantic-segmentation-with-output_32_0.png
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7c2084994369e17be27f2bc482a182c3b20d42564b4e2f818ca451e8c2ea6cea
-size 909654
diff --git a/docs/notebooks/segmind-vegart-with-output.rst b/docs/notebooks/segmind-vegart-with-output.rst
index 45266e95e9e..f94309ad55f 100644
--- a/docs/notebooks/segmind-vegart-with-output.rst
+++ b/docs/notebooks/segmind-vegart-with-output.rst
@@ -110,17 +110,17 @@ pipeline on disk.
from diffusers import LCMScheduler, AutoPipelineForText2Image
import gc
from pathlib import Path
-
+
model_id = "segmind/Segmind-Vega"
adapter_id = "segmind/Segmind-VegaRT"
pt_model_dir = Path("segmind-vegart")
-
+
if not pt_model_dir.exists():
pipe = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights(adapter_id)
pipe.fuse_lora()
-
+
pipe.save_pretrained("segmind-vegart")
del pipe
gc.collect()
@@ -176,7 +176,7 @@ back to image format.
.. code:: ipython3
from pathlib import Path
-
+
model_dir = Path("openvino-segmind-vegart")
sdxl_model_id = "./segmind-vegart"
tae_id = "madebyollin/taesdxl"
@@ -188,26 +188,26 @@ back to image format.
import openvino as ov
from diffusers import AutoencoderTiny
import gc
-
-
+
+
class VAEEncoder(torch.nn.Module):
def __init__(self, vae):
super().__init__()
self.vae = vae
-
+
def forward(self, sample):
return self.vae.encode(sample)
-
-
+
+
class VAEDecoder(torch.nn.Module):
def __init__(self, vae):
super().__init__()
self.vae = vae
-
+
def forward(self, latent_sample):
return self.vae.decode(latent_sample)
-
-
+
+
def convert_tiny_vae(model_id, output_path):
tiny_vae = AutoencoderTiny.from_pretrained(model_id)
tiny_vae.eval()
@@ -222,8 +222,8 @@ back to image format.
del tiny_vae
del ov_model
gc.collect()
-
-
+
+
if not skip_convert_model:
!optimum-cli export openvino --model $sdxl_model_id --task stable-diffusion-xl $model_dir --fp16
convert_tiny_vae(tae_id, model_dir)
@@ -245,7 +245,7 @@ For saving time, we will not cover image-to-image generation in this
notebook. As we already mentioned, Segmind-Vega is compatible with
Stable Diffusion XL pipeline, the steps required to run Stable Diffusion
XL inference for image-to-image task were discussed in this
-`notebook `__.
+`notebook `__.
Select inference device for text-to-image generation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -255,16 +255,16 @@ Select inference device for text-to-image generation
.. code:: ipython3
import ipywidgets as widgets
-
+
core = ov.Core()
-
+
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="AUTO",
description="Device:",
disabled=False,
)
-
+
device
@@ -279,7 +279,7 @@ Select inference device for text-to-image generation
.. code:: ipython3
from optimum.intel.openvino import OVStableDiffusionXLPipeline
-
+
text2image_pipe = OVStableDiffusionXLPipeline.from_pretrained(model_dir, device=device.value)
@@ -301,9 +301,9 @@ Select inference device for text-to-image generation
.. code:: ipython3
from transformers import set_seed
-
+
set_seed(23)
-
+
prompt = "A cinematic highly detailed shot of a baby Yorkshire terrier wearing an intricate Italian priest robe, with crown"
image = text2image_pipe(prompt, num_inference_steps=4, height=512, width=512, guidance_scale=0.5).images[0]
image.save("dog.png")
@@ -363,7 +363,7 @@ improve model inference speed.
description="Quantization",
disabled=False,
)
-
+
to_quantize
@@ -379,14 +379,14 @@ improve model inference speed.
# Fetch `skip_kernel_extension` module
import requests
-
+
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
)
open("skip_kernel_extension.py", "w").write(r.text)
-
+
int8_pipe = None
-
+
if to_quantize.value and "GPU" in device.value:
to_quantize.value = False
42
@@ -405,8 +405,8 @@ model inputs for calibration we should customize ``CompiledModel``.
.. code:: ipython3
UNET_INT8_OV_PATH = model_dir / "optimized_unet" / "openvino_model.xml"
-
-
+
+
def disable_progress_bar(pipeline, disable=True):
if not hasattr(pipeline, "_progress_bar_config"):
pipeline._progress_bar_config = {"disable": disable}
@@ -416,31 +416,31 @@ model inputs for calibration we should customize ``CompiledModel``.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
import datasets
import numpy as np
from tqdm.notebook import tqdm
from transformers import set_seed
from typing import Any, Dict, List
-
+
set_seed(1)
-
+
class CompiledModelDecorator(ov.CompiledModel):
def __init__(self, compiled_model: ov.CompiledModel, data_cache: List[Any] = None):
super().__init__(compiled_model)
self.data_cache = data_cache if data_cache else []
-
+
def __call__(self, *args, **kwargs):
self.data_cache.append(*args)
return super().__call__(*args, **kwargs)
-
+
def collect_calibration_data(pipe, subset_size: int) -> List[Dict]:
original_unet = pipe.unet.request
pipe.unet.request = CompiledModelDecorator(original_unet)
-
+
dataset = datasets.load_dataset("conceptual_captions", split="train").shuffle(seed=42)
disable_progress_bar(pipe)
-
+
# Run inference for data collection
pbar = tqdm(total=subset_size)
diff = 0
@@ -462,7 +462,7 @@ model inputs for calibration we should customize ``CompiledModel``.
break
pbar.update(collected_subset_size - diff)
diff = collected_subset_size
-
+
calibration_dataset = pipe.unet.request.data_cache
disable_progress_bar(pipe, disable=False)
pipe.unet.request = original_unet
@@ -471,7 +471,7 @@ model inputs for calibration we should customize ``CompiledModel``.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
if not UNET_INT8_OV_PATH.exists():
text2image_pipe = OVStableDiffusionXLPipeline.from_pretrained(model_dir, device=device.value)
unet_calibration_data = collect_calibration_data(text2image_pipe, subset_size=200)
@@ -492,10 +492,10 @@ sensitive ``Convolution`` layers in FP16 precision.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
import nncf
from nncf.scopes import IgnoredScope
-
+
UNET_OV_PATH = model_dir / "unet" / "openvino_model.xml"
if not UNET_INT8_OV_PATH.exists():
unet = core.read_model(UNET_OV_PATH)
@@ -516,7 +516,7 @@ sensitive ``Convolution`` layers in FP16 precision.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
def create_int8_pipe(model_dir, unet_int8_path, device, core, unet_device='CPU'):
int8_pipe = OVStableDiffusionXLPipeline.from_pretrained(model_dir, device=device, compile=True)
del int8_pipe.unet.request
@@ -525,12 +525,12 @@ sensitive ``Convolution`` layers in FP16 precision.
int8_pipe.unet.model = core.read_model(unet_int8_path)
int8_pipe.unet.request = core.compile_model(int8_pipe.unet.model, unet_device or device)
return int8_pipe
-
+
int8_text2image_pipe = create_int8_pipe(model_dir, UNET_INT8_OV_PATH, device.value, core)
-
-
+
+
set_seed(23)
-
+
image = int8_text2image_pipe(prompt, num_inference_steps=4, height=512, width=512, guidance_scale=0.5).images[0]
display(image)
@@ -563,10 +563,10 @@ Compare UNet file size
.. code:: ipython3
%%skip not $to_quantize.value
-
+
fp16_ir_model_size = UNET_OV_PATH.with_suffix(".bin").stat().st_size / 1024
quantized_model_size = UNET_INT8_OV_PATH.with_suffix(".bin").stat().st_size / 1024
-
+
print(f"FP16 model size: {fp16_ir_model_size:.2f} KB")
print(f"INT8 model size: {quantized_model_size:.2f} KB")
print(f"Model compression rate: {fp16_ir_model_size / quantized_model_size:.3f}")
@@ -594,9 +594,9 @@ pipelines, we use median inference time on the calibration subset.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
import time
-
+
validation_size = 7
calibration_dataset = datasets.load_dataset("conceptual_captions", split="train")
validation_data = []
@@ -605,11 +605,11 @@ pipelines, we use median inference time on the calibration subset.
break
prompt = batch["caption"]
validation_data.append(prompt)
-
+
def calculate_inference_time(pipe, dataset):
inference_time = []
disable_progress_bar(pipe)
-
+
for prompt in dataset:
start = time.perf_counter()
image = pipe(
@@ -634,15 +634,15 @@ pipelines, we use median inference time on the calibration subset.
.. code:: ipython3
%%skip not $to_quantize.value
-
+
int8_latency = calculate_inference_time(int8_text2image_pipe, validation_data)
-
+
del int8_text2image_pipe
gc.collect()
-
+
text2image_pipe = OVStableDiffusionXLPipeline.from_pretrained(model_dir, device=device.value)
fp_latency = calculate_inference_time(text2image_pipe, validation_data)
-
+
del text2image_pipe
gc.collect()
print(f"FP16 pipeline latency: {fp_latency:.3f}")
@@ -684,13 +684,13 @@ launch the interactive demo.
.. code:: ipython3
quantized_model_present = UNET_INT8_OV_PATH.exists()
-
+
use_quantized_model = widgets.Checkbox(
value=quantized_model_present,
description="Use quantized model",
disabled=not quantized_model_present,
)
-
+
use_quantized_model
@@ -705,16 +705,16 @@ launch the interactive demo.
.. code:: ipython3
import gradio as gr
-
+
if use_quantized_model.value:
if not quantized_model_present:
raise RuntimeError("Quantized model not found.")
text2image_pipe = create_int8_pipe(model_dir, UNET_INT8_OV_PATH, device.value, core)
-
+
else:
text2image_pipe = OVStableDiffusionXLPipeline.from_pretrained(model_dir, device=device.value)
-
-
+
+
def generate_from_text(text, seed, num_steps, height, width):
set_seed(seed)
result = text2image_pipe(
@@ -725,8 +725,8 @@ launch the interactive demo.
width=width,
).images[0]
return result
-
-
+
+
with gr.Blocks() as demo:
with gr.Column():
positive_input = gr.Textbox(label="Text prompt")
@@ -772,7 +772,7 @@ launch the interactive demo.
],
[positive_input, seed_input],
)
-
+
# if you are launching remotely, specify server_name and server_port
# demo.launch(server_name='your server name', server_port='server port in int')
# Read more in the docs: https://gradio.app/docs/
diff --git a/docs/notebooks/siglip-zero-shot-image-classification-with-output.rst b/docs/notebooks/siglip-zero-shot-image-classification-with-output.rst
index 8776a4e67a5..502c4265933 100644
--- a/docs/notebooks/siglip-zero-shot-image-classification-with-output.rst
+++ b/docs/notebooks/siglip-zero-shot-image-classification-with-output.rst
@@ -99,28 +99,11 @@ tokenizer and preparing the images.
%pip install -q "matplotlib>=3.4,<3.7"
-.. parsed-literal::
-
- WARNING: typer 0.12.3 does not provide the extra 'all'
-
-
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -134,14 +117,14 @@ tokenizer and preparing the images.
.. parsed-literal::
- 2024-04-18 00:44:12.841205: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:44:12.876405: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 01:10:53.448821: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 01:10:53.484554: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:44:13.472113: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-07 01:10:54.080932: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Run PyTorch model inference
@@ -277,19 +260,11 @@ object ready to load on the device and start making predictions.
.. parsed-literal::
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/siglip/modeling_siglip.py:362: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/siglip/modeling_siglip.py:354: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (batch_size, self.num_heads, q_len, k_v_seq_len):
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/siglip/modeling_siglip.py:380: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/siglip/modeling_siglip.py:372: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (batch_size, self.num_heads, q_len, self.head_dim):
@@ -495,7 +470,7 @@ model.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1461: FutureWarning: The repository for conceptual_captions contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/conceptual_captions
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1486: FutureWarning: The repository for conceptual_captions contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/conceptual_captions
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
warnings.warn(
@@ -672,7 +647,7 @@ model are similar to the PyTorch model.
.. parsed-literal::
- [{'dog': 0.99}, {'horse': 0.0}, {'cat': 0.0}, {'wolf': 0.0}, {'tiger': 0.0}]
+ [{'dog': 1.0}, {'horse': 0.0}, {'cat': 0.0}, {'tiger': 0.0}, {'wolf': 0.0}]
@@ -704,8 +679,8 @@ Compare File Size
.. parsed-literal::
FP16 IR model size: 387.49 MB
- INT8 model size: 196.46 MB
- Model compression rate: 1.972
+ INT8 model size: 201.26 MB
+ Model compression rate: 1.925
Compare inference time of the FP16 IR and quantized models
@@ -747,7 +722,7 @@ approximately estimate the speed up of the dynamic quantized models.
.. parsed-literal::
- Performance speed up: 2.117
+ Performance speed up: 2.092
Interactive inference
diff --git a/docs/notebooks/siglip-zero-shot-image-classification-with-output_files/siglip-zero-shot-image-classification-with-output_24_1.png b/docs/notebooks/siglip-zero-shot-image-classification-with-output_files/siglip-zero-shot-image-classification-with-output_24_1.png
index 156a28aacb6..c24903dd7d8 100644
--- a/docs/notebooks/siglip-zero-shot-image-classification-with-output_files/siglip-zero-shot-image-classification-with-output_24_1.png
+++ b/docs/notebooks/siglip-zero-shot-image-classification-with-output_files/siglip-zero-shot-image-classification-with-output_24_1.png
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
-oid sha256:beeadbab0dee7744db3c5541d8002ec04e28e12bd1236179940343143d2420d3
-size 580997
+oid sha256:a8a6f33d09e139009ae19dbf3aa4942040d04c4abf091e3a31213d6c140eb580
+size 580932
diff --git a/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output.rst b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output.rst
new file mode 100644
index 00000000000..604796e9729
--- /dev/null
+++ b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output.rst
@@ -0,0 +1,926 @@
+One Step Sketch to Image translation with pix2pix-turbo and OpenVINO
+====================================================================
+
+Diffusion models achieve remarkable results in image generation. They
+are able synthesize high-quality images guided by user instructions. In
+the same time, majority of diffusion-based image generation approaches
+are time-consuming due to the iterative denoising process.Pix2Pix-turbo
+model was proposed in `One-Step Image Translation with Text-to-Image
+Models paper `__ for addressing
+slowness of diffusion process in image-to-image translation task. It is
+based on `SD-Turbo `__, a
+fast generative text-to-image model that can synthesize photorealistic
+images from a text prompt in a single network evaluation. Using only
+single inference, pix2pix-turbo achieves comparable by quality results
+with recent works such as ControlNet for Sketch2Photo and Edge2Image for
+50 steps.
+
+|image0|
+
+In this tutorial you will learn how to turn sketches to images using
+`Pix2Pix-Turbo `__ and
+OpenVINO. #### Table of contents:
+
+- `Prerequisites <#prerequisites>`__
+- `Load PyTorch model <#load-pytorch-model>`__
+- `Convert PyTorch model to Openvino Intermediate Representation
+ format <#convert-pytorch-model-to-openvino-intermediate-representation-format>`__
+- `Select inference device <#select-inference-device>`__
+- `Compile model <#compile-model>`__
+- `Run model inference <#run-model-inference>`__
+- `Interactive demo <#interactive-demo>`__
+
+.. |image0| image:: https://github.com/GaParmar/img2img-turbo/raw/main/assets/gen_variations.jpg
+
+Prerequisites
+-------------
+
+
+
+Clone `model repository `__
+and install required packages.
+
+.. code:: ipython3
+
+ %pip install -q "openvino>=2024.1.0" "torch>=2.1" torchvision "diffusers==0.25.1" "peft==0.6.2" transformers tqdm pillow opencv-python "gradio==3.43.1" --extra-index-url https://download.pytorch.org/whl/cpu
+
+
+.. parsed-literal::
+
+ DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
+ Note: you may need to restart the kernel to use updated packages.
+
+
+.. code:: ipython3
+
+ from pathlib import Path
+
+ repo_dir = Path("img2img-turbo")
+
+ if not repo_dir.exists():
+ !git clone https://github.com/GaParmar/img2img-turbo.git
+
+ pix2pix_turbo_py_path = repo_dir / "src/pix2pix_turbo.py"
+ model_py_path = repo_dir / "src/model.py"
+ orig_pix2pix_turbo_path = pix2pix_turbo_py_path.parent / ("orig_" + pix2pix_turbo_py_path.name)
+ orig_model_py_path = model_py_path.parent / ("orig_" + model_py_path.name)
+
+ if not orig_pix2pix_turbo_path.exists():
+ pix2pix_turbo_py_path.rename(orig_pix2pix_turbo_path)
+
+ with orig_pix2pix_turbo_path.open("r") as f:
+ data = f.read()
+ data = data.replace("cuda", "cpu")
+ with pix2pix_turbo_py_path.open("w") as out_f:
+ out_f.write(data)
+
+ if not orig_model_py_path.exists():
+ model_py_path.rename(orig_model_py_path)
+
+ with orig_model_py_path.open("r") as f:
+ data = f.read()
+ data = data.replace("cuda", "cpu")
+ with model_py_path.open("w") as out_f:
+ out_f.write(data)
+ %cd $repo_dir
+
+
+.. parsed-literal::
+
+ Cloning into 'img2img-turbo'...
+ remote: Enumerating objects: 205, done.[K
+ remote: Counting objects: 100% (70/70), done.[K
+ remote: Compressing objects: 100% (26/26), done.[K
+ remote: Total 205 (delta 53), reused 44 (delta 44), pack-reused 135[K
+ Receiving objects: 100% (205/205), 31.89 MiB | 8.08 MiB/s, done.
+ Resolving deltas: 100% (96/96), done.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/sketch-to-image-pix2pix-turbo/img2img-turbo
+
+
+Load PyTorch model
+------------------
+
+
+
+Pix2Pix-turbo architecture illustrated on the diagram below. Model
+combines three separate modules in the original latent diffusion models
+into a single end-to-end network with small trainable weights. This
+architecture allows translation the input image x to the output y, while
+retaining the input scene structure. Authors use LoRA adapters in each
+module, introduce skip connections and Zero-Convolutions between input
+and output, and retrain the first layer of the U-Net. Blue boxes on
+diagram indicate trainable layers. Semi-transparent layers are frozen.
+|model_diagram|
+
+.. |model_diagram| image:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/18f1a442-8547-4edd-85b0-d8bd1a99bdf1
+
+.. code:: ipython3
+
+ import requests
+ import copy
+ from tqdm import tqdm
+ import torch
+ from transformers import AutoTokenizer, CLIPTextModel
+ from diffusers import AutoencoderKL, UNet2DConditionModel
+ from diffusers.models.autoencoders.vae import DiagonalGaussianDistribution
+ from diffusers.utils.peft_utils import set_weights_and_activate_adapters
+ from peft import LoraConfig
+ import types
+
+ from src.model import make_1step_sched
+ from src.pix2pix_turbo import TwinConv
+
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/sd-turbo", subfolder="tokenizer")
+
+
+ def tokenize_prompt(prompt):
+ caption_tokens = tokenizer(prompt, max_length=tokenizer.model_max_length, padding="max_length", truncation=True, return_tensors="pt").input_ids
+ return caption_tokens
+
+
+ def _vae_encoder_fwd(self, sample):
+ sample = self.conv_in(sample)
+ l_blocks = []
+ # down
+ for down_block in self.down_blocks:
+ l_blocks.append(sample)
+ sample = down_block(sample)
+ # middle
+ sample = self.mid_block(sample)
+ sample = self.conv_norm_out(sample)
+ sample = self.conv_act(sample)
+ sample = self.conv_out(sample)
+ current_down_blocks = l_blocks
+ return sample, current_down_blocks
+
+
+ def _vae_decoder_fwd(self, sample, incoming_skip_acts, latent_embeds=None):
+ sample = self.conv_in(sample)
+ upscale_dtype = next(iter(self.up_blocks.parameters())).dtype
+ # middle
+ sample = self.mid_block(sample, latent_embeds)
+ sample = sample.to(upscale_dtype)
+ if not self.ignore_skip:
+ skip_convs = [self.skip_conv_1, self.skip_conv_2, self.skip_conv_3, self.skip_conv_4]
+ # up
+ for idx, up_block in enumerate(self.up_blocks):
+ skip_in = skip_convs[idx](incoming_skip_acts[::-1][idx] * self.gamma)
+ # add skip
+ sample = sample + skip_in
+ sample = up_block(sample, latent_embeds)
+ else:
+ for idx, up_block in enumerate(self.up_blocks):
+ sample = up_block(sample, latent_embeds)
+ # post-process
+ if latent_embeds is None:
+ sample = self.conv_norm_out(sample)
+ else:
+ sample = self.conv_norm_out(sample, latent_embeds)
+ sample = self.conv_act(sample)
+ sample = self.conv_out(sample)
+ return sample
+
+
+ def vae_encode(self, x: torch.FloatTensor):
+ """
+ Encode a batch of images into latents.
+
+ Args:
+ x (`torch.FloatTensor`): Input batch of images.
+
+ Returns:
+ The latent representations of the encoded images. If `return_dict` is True, a
+ [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned.
+ """
+ h, down_blocks = self.encoder(x)
+
+ moments = self.quant_conv(h)
+ posterior = DiagonalGaussianDistribution(moments)
+
+ return (posterior, down_blocks)
+
+
+ def vae_decode(self, z: torch.FloatTensor, skip_acts):
+ decoded = self._decode(z, skip_acts)[0]
+ return (decoded,)
+
+
+ def vae__decode(self, z: torch.FloatTensor, skip_acts):
+ z = self.post_quant_conv(z)
+ dec = self.decoder(z, skip_acts)
+
+ return (dec,)
+
+
+ class Pix2PixTurbo(torch.nn.Module):
+ def __init__(self, pretrained_name=None, pretrained_path=None, ckpt_folder="checkpoints", lora_rank_unet=8, lora_rank_vae=4):
+ super().__init__()
+ self.text_encoder = CLIPTextModel.from_pretrained("stabilityai/sd-turbo", subfolder="text_encoder").cpu()
+ self.sched = make_1step_sched()
+
+ vae = AutoencoderKL.from_pretrained("stabilityai/sd-turbo", subfolder="vae")
+ vae.encoder.forward = types.MethodType(_vae_encoder_fwd, vae.encoder)
+ vae.decoder.forward = types.MethodType(_vae_decoder_fwd, vae.decoder)
+ vae.encode = types.MethodType(vae_encode, vae)
+ vae.decode = types.MethodType(vae_decode, vae)
+ vae._decode = types.MethodType(vae__decode, vae)
+ # add the skip connection convs
+ vae.decoder.skip_conv_1 = torch.nn.Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False).cpu()
+ vae.decoder.skip_conv_2 = torch.nn.Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False).cpu()
+ vae.decoder.skip_conv_3 = torch.nn.Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False).cpu()
+ vae.decoder.skip_conv_4 = torch.nn.Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False).cpu()
+ vae.decoder.ignore_skip = False
+ unet = UNet2DConditionModel.from_pretrained("stabilityai/sd-turbo", subfolder="unet")
+ ckpt_folder = Path(ckpt_folder)
+
+ if pretrained_name == "edge_to_image":
+ url = "https://www.cs.cmu.edu/~img2img-turbo/models/edge_to_image_loras.pkl"
+ ckpt_folder.mkdir(exist_ok=True)
+ outf = ckpt_folder / "edge_to_image_loras.pkl"
+ if not outf:
+ print(f"Downloading checkpoint to {outf}")
+ response = requests.get(url, stream=True)
+ total_size_in_bytes = int(response.headers.get("content-length", 0))
+ block_size = 1024 # 1 Kibibyte
+ progress_bar = tqdm(total=total_size_in_bytes, unit="iB", unit_scale=True)
+ with open(outf, "wb") as file:
+ for data in response.iter_content(block_size):
+ progress_bar.update(len(data))
+ file.write(data)
+ progress_bar.close()
+ if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes:
+ print("ERROR, something went wrong")
+ print(f"Downloaded successfully to {outf}")
+ p_ckpt = outf
+ sd = torch.load(p_ckpt, map_location="cpu")
+ unet_lora_config = LoraConfig(r=sd["rank_unet"], init_lora_weights="gaussian", target_modules=sd["unet_lora_target_modules"])
+ vae_lora_config = LoraConfig(r=sd["rank_vae"], init_lora_weights="gaussian", target_modules=sd["vae_lora_target_modules"])
+ vae.add_adapter(vae_lora_config, adapter_name="vae_skip")
+ _sd_vae = vae.state_dict()
+ for k in sd["state_dict_vae"]:
+ _sd_vae[k] = sd["state_dict_vae"][k]
+ vae.load_state_dict(_sd_vae)
+ unet.add_adapter(unet_lora_config)
+ _sd_unet = unet.state_dict()
+ for k in sd["state_dict_unet"]:
+ _sd_unet[k] = sd["state_dict_unet"][k]
+ unet.load_state_dict(_sd_unet)
+
+ elif pretrained_name == "sketch_to_image_stochastic":
+ # download from url
+ url = "https://www.cs.cmu.edu/~img2img-turbo/models/sketch_to_image_stochastic_lora.pkl"
+ ckpt_folder.mkdir(exist_ok=True)
+ outf = ckpt_folder / "sketch_to_image_stochastic_lora.pkl"
+ if not outf.exists():
+ print(f"Downloading checkpoint to {outf}")
+ response = requests.get(url, stream=True)
+ total_size_in_bytes = int(response.headers.get("content-length", 0))
+ block_size = 1024 # 1 Kibibyte
+ progress_bar = tqdm(total=total_size_in_bytes, unit="iB", unit_scale=True)
+ with open(outf, "wb") as file:
+ for data in response.iter_content(block_size):
+ progress_bar.update(len(data))
+ file.write(data)
+ progress_bar.close()
+ if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes:
+ print("ERROR, something went wrong")
+ print(f"Downloaded successfully to {outf}")
+ p_ckpt = outf
+ convin_pretrained = copy.deepcopy(unet.conv_in)
+ unet.conv_in = TwinConv(convin_pretrained, unet.conv_in)
+ sd = torch.load(p_ckpt, map_location="cpu")
+ unet_lora_config = LoraConfig(r=sd["rank_unet"], init_lora_weights="gaussian", target_modules=sd["unet_lora_target_modules"])
+ vae_lora_config = LoraConfig(r=sd["rank_vae"], init_lora_weights="gaussian", target_modules=sd["vae_lora_target_modules"])
+ vae.add_adapter(vae_lora_config, adapter_name="vae_skip")
+ _sd_vae = vae.state_dict()
+ for k in sd["state_dict_vae"]:
+ if k not in _sd_vae:
+ continue
+ _sd_vae[k] = sd["state_dict_vae"][k]
+
+ vae.load_state_dict(_sd_vae)
+ unet.add_adapter(unet_lora_config)
+ _sd_unet = unet.state_dict()
+ for k in sd["state_dict_unet"]:
+ _sd_unet[k] = sd["state_dict_unet"][k]
+ unet.load_state_dict(_sd_unet)
+
+ elif pretrained_path is not None:
+ sd = torch.load(pretrained_path, map_location="cpu")
+ unet_lora_config = LoraConfig(r=sd["rank_unet"], init_lora_weights="gaussian", target_modules=sd["unet_lora_target_modules"])
+ vae_lora_config = LoraConfig(r=sd["rank_vae"], init_lora_weights="gaussian", target_modules=sd["vae_lora_target_modules"])
+ vae.add_adapter(vae_lora_config, adapter_name="vae_skip")
+ _sd_vae = vae.state_dict()
+ for k in sd["state_dict_vae"]:
+ _sd_vae[k] = sd["state_dict_vae"][k]
+ vae.load_state_dict(_sd_vae)
+ unet.add_adapter(unet_lora_config)
+ _sd_unet = unet.state_dict()
+ for k in sd["state_dict_unet"]:
+ _sd_unet[k] = sd["state_dict_unet"][k]
+ unet.load_state_dict(_sd_unet)
+
+ # unet.enable_xformers_memory_efficient_attention()
+ unet.to("cpu")
+ vae.to("cpu")
+ self.unet, self.vae = unet, vae
+ self.vae.decoder.gamma = 1
+ self.timesteps = torch.tensor([999], device="cpu").long()
+ self.text_encoder.requires_grad_(False)
+
+ def set_r(self, r):
+ self.unet.set_adapters(["default"], weights=[r])
+ set_weights_and_activate_adapters(self.vae, ["vae_skip"], [r])
+ self.r = r
+ self.unet.conv_in.r = r
+ self.vae.decoder.gamma = r
+
+ def forward(self, c_t, prompt_tokens, noise_map):
+ caption_enc = self.text_encoder(prompt_tokens)[0]
+ # scale the lora weights based on the r value
+ sample, current_down_blocks = self.vae.encode(c_t)
+ encoded_control = sample.sample() * self.vae.config.scaling_factor
+ # combine the input and noise
+ unet_input = encoded_control * self.r + noise_map * (1 - self.r)
+
+ unet_output = self.unet(
+ unet_input,
+ self.timesteps,
+ encoder_hidden_states=caption_enc,
+ ).sample
+ x_denoised = self.sched.step(unet_output, self.timesteps, unet_input, return_dict=True).prev_sample
+ output_image = (self.vae.decode(x_denoised / self.vae.config.scaling_factor, current_down_blocks)[0]).clamp(-1, 1)
+ return output_image
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
+ torch.utils._pytree._register_pytree_node(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+
+
+.. code:: ipython3
+
+ ov_model_path = Path("model/pix2pix-turbo.xml")
+
+ pt_model = None
+
+ if not ov_model_path.exists():
+ pt_model = Pix2PixTurbo("sketch_to_image_stochastic")
+ pt_model.set_r(0.4)
+ pt_model.eval()
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
+ torch.utils._pytree._register_pytree_node(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
+
+
+.. parsed-literal::
+
+ Downloading checkpoint to checkpoints/sketch_to_image_stochastic_lora.pkl
+
+
+.. parsed-literal::
+
+ 100%|██████████| 525M/525M [08:38<00:00, 1.01MiB/s]
+
+
+.. parsed-literal::
+
+ Downloaded successfully to checkpoints/sketch_to_image_stochastic_lora.pkl
+
+
+Convert PyTorch model to Openvino Intermediate Representation format
+--------------------------------------------------------------------
+
+
+
+Starting from OpenVINO 2023.0 release, OpenVINO supports direct PyTorch
+models conversion to `OpenVINO Intermediate Representation (IR)
+format `__
+to take the advantage of advanced OpenVINO optimization tools and
+features. You need to provide a model object, input data for model
+tracing to `OpenVINO Model Conversion
+API `__.
+``ov.convert_model`` function convert PyTorch model instance to
+``ov.Model`` object that can be used for compilation on device or saved
+on disk using ``ov.save_model`` in compressed to FP16 format.
+
+.. code:: ipython3
+
+ import gc
+ import openvino as ov
+
+ if not ov_model_path.exists():
+ example_input = [torch.ones((1, 3, 512, 512)), torch.ones([1, 77], dtype=torch.int64), torch.ones([1, 4, 64, 64])]
+ with torch.no_grad():
+ ov_model = ov.convert_model(pt_model, example_input=example_input, input=[[1, 3, 512, 512], [1, 77], [1, 4, 64, 64]])
+ ov.save_model(ov_model, ov_model_path)
+ del ov_model
+ torch._C._jit_clear_class_registry()
+ torch.jit._recursive.concrete_type_store = torch.jit._recursive.ConcreteTypeStore()
+ torch.jit._state._clear_class_state()
+ del pt_model
+ gc.collect();
+
+
+.. parsed-literal::
+
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:86: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if input_shape[-1] > 1 or self.sliding_window is not None:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:162: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if past_key_values_length > 0:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:279: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:319: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/downsampling.py:135: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/downsampling.py:144: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/unet_2d_condition.py:915: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if dim % default_overall_up_factor != 0:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ assert hidden_states.shape[1] == self.channels
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if hidden_states.shape[0] >= 64:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_ddpm.py:433: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if model_output.shape[1] == sample.shape[1] * 2 and self.variance_type in ["learned", "learned_range"]:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_ddpm.py:440: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ alpha_prod_t_prev = self.alphas_cumprod[prev_t] if prev_t >= 0 else self.one
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_ddpm.py:479: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ if t > 0:
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/schedulers/scheduling_ddpm.py:330: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ alpha_prod_t_prev = self.alphas_cumprod[prev_t] if prev_t >= 0 else self.one
+
+
+Select inference device
+-----------------------
+
+
+
+.. code:: ipython3
+
+ import ipywidgets as widgets
+
+ core = ov.Core()
+ device = widgets.Dropdown(
+ options=core.available_devices + ["AUTO"],
+ value="AUTO",
+ description="Device:",
+ disabled=False,
+ )
+
+ device
+
+
+
+
+.. parsed-literal::
+
+ Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
+
+
+
+Compile model
+-------------
+
+
+
+.. code:: ipython3
+
+ compiled_model = core.compile_model(ov_model_path, device.value)
+
+Run model inference
+-------------------
+
+
+
+Now, let’s try model in action and turn simple cat sketch into
+professional artwork.
+
+.. code:: ipython3
+
+ from diffusers.utils import load_image
+
+ sketch_image = load_image("https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/f964a51d-34e8-411a-98f4-5f97a28f56b0")
+
+ sketch_image
+
+
+
+
+.. image:: sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.png
+
+
+
+.. code:: ipython3
+
+ import torchvision.transforms.functional as F
+
+ torch.manual_seed(145)
+ c_t = torch.unsqueeze(F.to_tensor(sketch_image) > 0.5, 0)
+ noise = torch.randn((1, 4, 512 // 8, 512 // 8))
+
+.. code:: ipython3
+
+ prompt_template = "anime artwork {prompt} . anime style, key visual, vibrant, studio anime, highly detailed"
+ prompt = prompt_template.replace("{prompt}", "fluffy magic cat")
+
+ prompt_tokens = tokenize_prompt(prompt)
+
+.. code:: ipython3
+
+ result = compiled_model([1 - c_t.to(torch.float32), prompt_tokens, noise])[0]
+
+.. code:: ipython3
+
+ from PIL import Image
+ import numpy as np
+
+ image_tensor = (result[0] * 0.5 + 0.5) * 255
+ image = np.transpose(image_tensor, (1, 2, 0)).astype(np.uint8)
+ Image.fromarray(image)
+
+
+
+
+.. image:: sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.png
+
+
+
+Interactive demo
+----------------
+
+
+
+In this section, you can try model on own paintings.
+
+**Instructions:** \* Enter a text prompt (e.g. cat) \* Start sketching,
+using pencil and eraser buttons \* Change the image style using a style
+template \* Try different seeds to generate different results \*
+Download results using download button
+
+.. code:: ipython3
+
+ import random
+ import base64
+ from io import BytesIO
+ import gradio as gr
+
+ style_list = [
+ {
+ "name": "Cinematic",
+ "prompt": "cinematic still {prompt} . emotional, harmonious, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy",
+ },
+ {
+ "name": "3D Model",
+ "prompt": "professional 3d model {prompt} . octane render, highly detailed, volumetric, dramatic lighting",
+ },
+ {
+ "name": "Anime",
+ "prompt": "anime artwork {prompt} . anime style, key visual, vibrant, studio anime, highly detailed",
+ },
+ {
+ "name": "Digital Art",
+ "prompt": "concept art {prompt} . digital artwork, illustrative, painterly, matte painting, highly detailed",
+ },
+ {
+ "name": "Photographic",
+ "prompt": "cinematic photo {prompt} . 35mm photograph, film, bokeh, professional, 4k, highly detailed",
+ },
+ {
+ "name": "Pixel art",
+ "prompt": "pixel-art {prompt} . low-res, blocky, pixel art style, 8-bit graphics",
+ },
+ {
+ "name": "Fantasy art",
+ "prompt": "ethereal fantasy concept art of {prompt} . magnificent, celestial, ethereal, painterly, epic, majestic, magical, fantasy art, cover art, dreamy",
+ },
+ {
+ "name": "Neonpunk",
+ "prompt": "neonpunk style {prompt} . cyberpunk, vaporwave, neon, vibes, vibrant, stunningly beautiful, crisp, detailed, sleek, ultramodern, magenta highlights, dark purple shadows, high contrast, cinematic, ultra detailed, intricate, professional",
+ },
+ {
+ "name": "Manga",
+ "prompt": "manga style {prompt} . vibrant, high-energy, detailed, iconic, Japanese comic style",
+ },
+ ]
+
+ styles = {k["name"]: k["prompt"] for k in style_list}
+ STYLE_NAMES = list(styles.keys())
+ DEFAULT_STYLE_NAME = "Fantasy art"
+ MAX_SEED = np.iinfo(np.int32).max
+
+
+ def pil_image_to_data_uri(img, format="PNG"):
+ buffered = BytesIO()
+ img.save(buffered, format=format)
+ img_str = base64.b64encode(buffered.getvalue()).decode()
+ return f"data:image/{format.lower()};base64,{img_str}"
+
+
+ def run(image, prompt, prompt_template, style_name, seed):
+ print(f"prompt: {prompt}")
+ print("sketch updated")
+ if image is None:
+ ones = Image.new("L", (512, 512), 255)
+ temp_uri = pil_image_to_data_uri(ones)
+ return ones, gr.update(link=temp_uri), gr.update(link=temp_uri)
+ prompt = prompt_template.replace("{prompt}", prompt)
+ image = image.convert("RGB")
+ image_t = F.to_tensor(image) > 0.5
+ print(f"seed={seed}")
+ caption_tokens = tokenizer(prompt, max_length=tokenizer.model_max_length, padding="max_length", truncation=True, return_tensors="pt").input_ids.cpu()
+ with torch.no_grad():
+ c_t = image_t.unsqueeze(0)
+ torch.manual_seed(seed)
+ B, C, H, W = c_t.shape
+ noise = torch.randn((1, 4, H // 8, W // 8))
+ output_image = torch.from_numpy(compiled_model([c_t.to(torch.float32), caption_tokens, noise])[0])
+ output_pil = F.to_pil_image(output_image[0].cpu() * 0.5 + 0.5)
+ input_sketch_uri = pil_image_to_data_uri(Image.fromarray(255 - np.array(image)))
+ output_image_uri = pil_image_to_data_uri(output_pil)
+ return (
+ output_pil,
+ gr.update(link=input_sketch_uri),
+ gr.update(link=output_image_uri),
+ )
+
+
+ def update_canvas(use_line, use_eraser):
+ if use_eraser:
+ _color = "#ffffff"
+ brush_size = 20
+ if use_line:
+ _color = "#000000"
+ brush_size = 4
+ return gr.update(brush_radius=brush_size, brush_color=_color, interactive=True)
+
+
+ def upload_sketch(file):
+ _img = Image.open(file.name)
+ _img = _img.convert("L")
+ return gr.update(value=_img, source="upload", interactive=True)
+
+
+ scripts = """
+ async () => {
+ globalThis.theSketchDownloadFunction = () => {
+ console.log("test")
+ var link = document.createElement("a");
+ dataUri = document.getElementById('download_sketch').href
+ link.setAttribute("href", dataUri)
+ link.setAttribute("download", "sketch.png")
+ document.body.appendChild(link); // Required for Firefox
+ link.click();
+ document.body.removeChild(link); // Clean up
+
+ // also call the output download function
+ theOutputDownloadFunction();
+ return false
+ }
+
+ globalThis.theOutputDownloadFunction = () => {
+ console.log("test output download function")
+ var link = document.createElement("a");
+ dataUri = document.getElementById('download_output').href
+ link.setAttribute("href", dataUri);
+ link.setAttribute("download", "output.png");
+ document.body.appendChild(link); // Required for Firefox
+ link.click();
+ document.body.removeChild(link); // Clean up
+ return false
+ }
+
+ globalThis.UNDO_SKETCH_FUNCTION = () => {
+ console.log("undo sketch function")
+ var button_undo = document.querySelector('#input_image > div.image-container.svelte-p3y7hu > div.svelte-s6ybro > button:nth-child(1)');
+ // Create a new 'click' event
+ var event = new MouseEvent('click', {
+ 'view': window,
+ 'bubbles': true,
+ 'cancelable': true
+ });
+ button_undo.dispatchEvent(event);
+ }
+
+ globalThis.DELETE_SKETCH_FUNCTION = () => {
+ console.log("delete sketch function")
+ var button_del = document.querySelector('#input_image > div.image-container.svelte-p3y7hu > div.svelte-s6ybro > button:nth-child(2)');
+ // Create a new 'click' event
+ var event = new MouseEvent('click', {
+ 'view': window,
+ 'bubbles': true,
+ 'cancelable': true
+ });
+ button_del.dispatchEvent(event);
+ }
+
+ globalThis.togglePencil = () => {
+ el_pencil = document.getElementById('my-toggle-pencil');
+ el_pencil.classList.toggle('clicked');
+ // simulate a click on the gradio button
+ btn_gradio = document.querySelector("#cb-line > label > input");
+ var event = new MouseEvent('click', {
+ 'view': window,
+ 'bubbles': true,
+ 'cancelable': true
+ });
+ btn_gradio.dispatchEvent(event);
+ if (el_pencil.classList.contains('clicked')) {
+ document.getElementById('my-toggle-eraser').classList.remove('clicked');
+ document.getElementById('my-div-pencil').style.backgroundColor = "gray";
+ document.getElementById('my-div-eraser').style.backgroundColor = "white";
+ }
+ else {
+ document.getElementById('my-toggle-eraser').classList.add('clicked');
+ document.getElementById('my-div-pencil').style.backgroundColor = "white";
+ document.getElementById('my-div-eraser').style.backgroundColor = "gray";
+ }
+ }
+
+ globalThis.toggleEraser = () => {
+ element = document.getElementById('my-toggle-eraser');
+ element.classList.toggle('clicked');
+ // simulate a click on the gradio button
+ btn_gradio = document.querySelector("#cb-eraser > label > input");
+ var event = new MouseEvent('click', {
+ 'view': window,
+ 'bubbles': true,
+ 'cancelable': true
+ });
+ btn_gradio.dispatchEvent(event);
+ if (element.classList.contains('clicked')) {
+ document.getElementById('my-toggle-pencil').classList.remove('clicked');
+ document.getElementById('my-div-pencil').style.backgroundColor = "white";
+ document.getElementById('my-div-eraser').style.backgroundColor = "gray";
+ }
+ else {
+ document.getElementById('my-toggle-pencil').classList.add('clicked');
+ document.getElementById('my-div-pencil').style.backgroundColor = "gray";
+ document.getElementById('my-div-eraser').style.backgroundColor = "white";
+ }
+ }
+ }
+ """
+
+ with gr.Blocks(css="style.css") as demo:
+ # these are hidden buttons that are used to trigger the canvas changes
+ line = gr.Checkbox(label="line", value=False, elem_id="cb-line")
+ eraser = gr.Checkbox(label="eraser", value=False, elem_id="cb-eraser")
+ with gr.Row(elem_id="main_row"):
+ with gr.Column(elem_id="column_input"):
+ gr.Markdown("## INPUT", elem_id="input_header")
+ image = gr.Image(
+ source="canvas",
+ tool="color-sketch",
+ type="pil",
+ image_mode="L",
+ invert_colors=True,
+ shape=(512, 512),
+ brush_radius=4,
+ height=440,
+ width=440,
+ brush_color="#000000",
+ interactive=True,
+ show_download_button=True,
+ elem_id="input_image",
+ show_label=False,
+ )
+ download_sketch = gr.Button("Download sketch", scale=1, elem_id="download_sketch")
+
+ gr.HTML(
+ """
+
+
+
+
+
+
+
+ """
+ )
+ # gr.Markdown("## Prompt", elem_id="tools_header")
+ prompt = gr.Textbox(label="Prompt", value="", show_label=True)
+ with gr.Row():
+ style = gr.Dropdown(
+ label="Style",
+ choices=STYLE_NAMES,
+ value=DEFAULT_STYLE_NAME,
+ scale=1,
+ )
+ prompt_temp = gr.Textbox(
+ label="Prompt Style Template",
+ value=styles[DEFAULT_STYLE_NAME],
+ scale=2,
+ max_lines=1,
+ )
+
+ with gr.Row():
+ seed = gr.Textbox(label="Seed", value=42, scale=1, min_width=50)
+ randomize_seed = gr.Button("Random", scale=1, min_width=50)
+
+ with gr.Column(elem_id="column_process", min_width=50, scale=0.4):
+ gr.Markdown("## pix2pix-turbo", elem_id="description")
+ run_button = gr.Button("Run", min_width=50)
+
+ with gr.Column(elem_id="column_output"):
+ gr.Markdown("## OUTPUT", elem_id="output_header")
+ result = gr.Image(
+ label="Result",
+ height=440,
+ width=440,
+ elem_id="output_image",
+ show_label=False,
+ show_download_button=True,
+ )
+ download_output = gr.Button("Download output", elem_id="download_output")
+ gr.Markdown("### Instructions")
+ gr.Markdown("**1**. Enter a text prompt (e.g. cat)")
+ gr.Markdown("**2**. Start sketching")
+ gr.Markdown("**3**. Change the image style using a style template")
+ gr.Markdown("**4**. Try different seeds to generate different results")
+
+ eraser.change(
+ fn=lambda x: gr.update(value=not x),
+ inputs=[eraser],
+ outputs=[line],
+ queue=False,
+ api_name=False,
+ ).then(update_canvas, [line, eraser], [image])
+ line.change(
+ fn=lambda x: gr.update(value=not x),
+ inputs=[line],
+ outputs=[eraser],
+ queue=False,
+ api_name=False,
+ ).then(update_canvas, [line, eraser], [image])
+
+ demo.load(None, None, None, _js=scripts)
+ randomize_seed.click(
+ lambda x: random.randint(0, MAX_SEED),
+ inputs=[],
+ outputs=seed,
+ queue=False,
+ api_name=False,
+ )
+ inputs = [image, prompt, prompt_temp, style, seed]
+ outputs = [result, download_sketch, download_output]
+ prompt.submit(fn=run, inputs=inputs, outputs=outputs, api_name=False)
+ style.change(
+ lambda x: styles[x],
+ inputs=[style],
+ outputs=[prompt_temp],
+ queue=False,
+ api_name=False,
+ ).then(
+ fn=run,
+ inputs=inputs,
+ outputs=outputs,
+ api_name=False,
+ )
+ run_button.click(fn=run, inputs=inputs, outputs=outputs, api_name=False)
+ image.change(run, inputs=inputs, outputs=outputs, queue=False, api_name=False)
+
+ try:
+ demo.queue().launch(debug=False)
+ except Exception:
+ demo.queue().launch(debug=False, share=True)
+ # if you are launching remotely, specify server_name and server_port
+ # demo.launch(server_name='your server name', server_port='server port in int')
+ # Read more in the docs: https://gradio.app/docs/
+
+
+.. parsed-literal::
+
+ /tmp/ipykernel_2886802/1555011934.py:259: GradioDeprecationWarning: 'scale' value should be an integer. Using 0.4 will cause issues.
+ with gr.Column(elem_id="column_process", min_width=50, scale=0.4):
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/gradio/utils.py:776: UserWarning: Expected 1 arguments for function at 0x7f72861f9dc0>, received 0.
+ warnings.warn(
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/gradio/utils.py:780: UserWarning: Expected at least 1 arguments for function at 0x7f72861f9dc0>, received 0.
+ warnings.warn(
+
+
+.. parsed-literal::
+
+ Running on local URL: http://127.0.0.1:7860
+
+ To create a public link, set `share=True` in `launch()`.
+
+
+
+
+
+
+
diff --git a/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.jpg b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.jpg
new file mode 100644
index 00000000000..d44930d1295
--- /dev/null
+++ b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.jpg
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecc12808bb01e9203f487cfacfb487e613b945d428c0c8b8bd4543b320645058
+size 12027
diff --git a/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.png b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.png
new file mode 100644
index 00000000000..f9551560338
--- /dev/null
+++ b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_14_0.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64c9d35bc76286eaab5acf7b3449a599139ccae9533be2f39bc61ff9dc127522
+size 27834
diff --git a/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.jpg b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.jpg
new file mode 100644
index 00000000000..5e7815f807d
--- /dev/null
+++ b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.jpg
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f98dd0a0866fe24d9076f351114c7f1810169a0c0c3dc87d19ffbf71e97da7ff
+size 23769
diff --git a/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.png b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.png
new file mode 100644
index 00000000000..3232fa7f47f
--- /dev/null
+++ b/docs/notebooks/sketch-to-image-pix2pix-turbo-with-output_files/sketch-to-image-pix2pix-turbo-with-output_18_0.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0de5c07b96eee9f36602fe7d349586b18c91cab0e6d6b8a4c16e0be88f8ab19e
+size 316127
diff --git a/docs/notebooks/slowfast-video-recognition-with-output.rst b/docs/notebooks/slowfast-video-recognition-with-output.rst
deleted file mode 100644
index 37e78430e33..00000000000
--- a/docs/notebooks/slowfast-video-recognition-with-output.rst
+++ /dev/null
@@ -1,838 +0,0 @@
-Video Recognition using SlowFast and OpenVINO™
-==============================================
-
-Teaching machines to detect, understand and analyze the contents of
-images has been one of the more well-known and well-studied problems in
-computer vision. However, analyzing videos to understand what is
-happening in them and detecting objects of interest are equally
-important and challenging tasks that have widespread applications in
-several areas including autonomous driving, healthcare, security, and
-many more.
-
-The SlowFast model puts forth an interesting approach to analyzing
-videos based on the intuition that videos typically contain static as
-well as dynamic elements- use a slow pathway operating at a low frame
-rate to analyze the static content and a fast pathway operating at a
-high frame rate to capture dynamic content. Its strength lies in its
-ability to effectively capture both fast and slow-motion information in
-video sequences, making it particularly well-suited to tasks that
-require a temporal and spatial understanding of the data.
-
-|image0|
-
-More details about the network can be found in the original
-`paper `__
-and `repository `__.
-
-In this notebook, we will see how to convert and run a ResNet-50 based
-SlowFast model using OpenVINO.
-
-This tutorial consists of the following steps
-
-- Preparing the PyTorch model
-- Download and prepare data
-- Check inference with the PyTorch model
-- Convert Model to OpenVINO Intermediate Representation
-- Verify inference with the converted model
-
-.. |image0| image:: https://user-images.githubusercontent.com/34324155/143044111-94676f64-7ba8-4081-9011-f8054bed7030.png
-
-Table of contents:
-^^^^^^^^^^^^^^^^^^
-
-- `Prepare PyTorch Model <#prepare-pytorch-model>`__
-
- - `Install necessary packages <#install-necessary-packages>`__
- - `Imports and Settings <#imports-and-settings>`__
-
-- `Convert model to OpenVINO Intermediate
- Representation <#convert-model-to-openvino-intermediate-representation>`__
-- `Select inference device <#select-inference-device>`__
-- `Verify Model Inference <#verify-model-inference>`__
-
-Prepare PyTorch Model
----------------------
-
-
-
-Install necessary packages
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- %pip install -q "openvino>=2023.3.0" torch opencv-python fvcore tqdm --extra-index-url https://download.pytorch.org/whl/cpu
-
-
-.. parsed-literal::
-
- DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
- Note: you may need to restart the kernel to use updated packages.
-
-
-Imports and Settings
-~~~~~~~~~~~~~~~~~~~~
-
-
-
-.. code:: ipython3
-
- import json
- import math
- import cv2
- import torch
- import numpy as np
- from pathlib import Path
- from typing import Any, List, Dict
- from IPython.display import Video
- import openvino as ov
-
- # Fetch `notebook_utils` module
- import requests
-
- r = requests.get(
- url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
- )
-
- open("notebook_utils.py", "w").write(r.text)
- from notebook_utils import download_file
-
-.. code:: ipython3
-
- DATA_DIR = Path("data/")
- MODEL_DIR = Path("model/")
- MODEL_DIR.mkdir(exist_ok=True)
- DATA_DIR.mkdir(exist_ok=True)
-
-To begin, we download the PyTorch model from the PyTorchVideo
-repository. In this notebook, we will be using a SlowFast Network based
-on the ResNet-50 architecture trained on the Kinetics 400 dataset.
-Kinetics 400 is a large-scale dataset for action recognition in videos,
-containing 400 human action classes, with at least 400 video clips for
-each action. Read more about the dataset and the paper
-`here `__.
-
-.. code:: ipython3
-
- MODEL_NAME = "slowfast_r50"
- MODEL_REPOSITORY = "facebookresearch/pytorchvideo"
- DEVICE = "cpu"
-
- # load the pretrained model from the repository
- model = torch.hub.load(
- repo_or_dir=MODEL_REPOSITORY,
- model=MODEL_NAME,
- pretrained=True,
- skip_validation=True,
- )
-
- # set the device to allocate tensors to. for example, "cpu" or "cuda"
- model.to(DEVICE)
-
- # set the model to eval mode
- model.eval()
-
-
-.. parsed-literal::
-
- Using cache found in /opt/home/k8sworker/.cache/torch/hub/facebookresearch_pytorchvideo_main
-
-
-
-
-.. parsed-literal::
-
- Net(
- (blocks): ModuleList(
- (0): MultiPathWayWithFuse(
- (multipathway_blocks): ModuleList(
- (0): ResNetBasicStem(
- (conv): Conv3d(3, 64, kernel_size=(1, 7, 7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
- (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- (pool): MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=[0, 1, 1], dilation=1, ceil_mode=False)
- )
- (1): ResNetBasicStem(
- (conv): Conv3d(3, 8, kernel_size=(5, 7, 7), stride=(1, 2, 2), padding=(2, 3, 3), bias=False)
- (norm): BatchNorm3d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- (pool): MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=[0, 1, 1], dilation=1, ceil_mode=False)
- )
- )
- (multipathway_fusion): FuseFastToSlow(
- (conv_fast_to_slow): Conv3d(8, 16, kernel_size=(7, 1, 1), stride=(4, 1, 1), padding=(3, 0, 0), bias=False)
- (norm): BatchNorm3d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- )
- )
- (1): MultiPathWayWithFuse(
- (multipathway_blocks): ModuleList(
- (0): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(80, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (branch1_norm): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(80, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-2): 2 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(256, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- (1): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(8, 32, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (branch1_norm): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(8, 8, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(8, 8, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(8, 32, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-2): 2 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(32, 8, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(8, 8, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(8, 32, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- )
- (multipathway_fusion): FuseFastToSlow(
- (conv_fast_to_slow): Conv3d(32, 64, kernel_size=(7, 1, 1), stride=(4, 1, 1), padding=(3, 0, 0), bias=False)
- (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- )
- )
- (2): MultiPathWayWithFuse(
- (multipathway_blocks): ModuleList(
- (0): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(320, 512, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(320, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-3): 3 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(512, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- (1): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(32, 64, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(32, 16, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(16, 16, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(16, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-3): 3 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(64, 16, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(16, 16, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(16, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- )
- (multipathway_fusion): FuseFastToSlow(
- (conv_fast_to_slow): Conv3d(64, 128, kernel_size=(7, 1, 1), stride=(4, 1, 1), padding=(3, 0, 0), bias=False)
- (norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- )
- )
- (3): MultiPathWayWithFuse(
- (multipathway_blocks): ModuleList(
- (0): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(640, 1024, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(640, 256, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(256, 256, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(256, 1024, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-5): 5 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(1024, 256, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(256, 256, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(256, 1024, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- (1): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(64, 128, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(64, 32, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(32, 32, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(32, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-5): 5 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(128, 32, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(32, 32, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(32, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- )
- (multipathway_fusion): FuseFastToSlow(
- (conv_fast_to_slow): Conv3d(128, 256, kernel_size=(7, 1, 1), stride=(4, 1, 1), padding=(3, 0, 0), bias=False)
- (norm): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (activation): ReLU()
- )
- )
- (4): MultiPathWayWithFuse(
- (multipathway_blocks): ModuleList(
- (0): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(1280, 2048, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(1280, 512, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(512, 512, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(512, 2048, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-2): 2 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(2048, 512, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(512, 512, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(512, 2048, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- (1): ResStage(
- (res_blocks): ModuleList(
- (0): ResBlock(
- (branch1_conv): Conv3d(128, 256, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
- (branch1_norm): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(128, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- (1-2): 2 x ResBlock(
- (branch2): BottleneckBlock(
- (conv_a): Conv3d(256, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
- (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_a): ReLU()
- (conv_b): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
- (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- (act_b): ReLU()
- (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
- (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
- )
- (activation): ReLU()
- )
- )
- )
- )
- (multipathway_fusion): Identity()
- )
- (5): PoolConcatPathway(
- (pool): ModuleList(
- (0): AvgPool3d(kernel_size=(8, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
- (1): AvgPool3d(kernel_size=(32, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
- )
- )
- (6): ResNetBasicHead(
- (dropout): Dropout(p=0.5, inplace=False)
- (proj): Linear(in_features=2304, out_features=400, bias=True)
- (output_pool): AdaptiveAvgPool3d(output_size=1)
- )
- )
- )
-
-
-
-Now that we have loaded our pre-trained model, we will check inference
-with it. Since the model returns the detected class IDs, we download the
-ID to class label mapping for the Kinetics 400 dataset and load the
-mapping to a dict for later use.
-
-.. code:: ipython3
-
- CLASSNAMES_SOURCE = "https://dl.fbaipublicfiles.com/pyslowfast/dataset/class_names/kinetics_classnames.json"
- CLASSNAMES_FILE = "kinetics_classnames.json"
-
- download_file(url=CLASSNAMES_SOURCE, directory=DATA_DIR, show_progress=True)
-
- # load from json
- with open(DATA_DIR / CLASSNAMES_FILE, "r") as f:
- kinetics_classnames = json.load(f)
-
- # load dict of id to class label mapping
- kinetics_id_to_classname = {}
- for k, v in kinetics_classnames.items():
- kinetics_id_to_classname[v] = str(k).replace('"', "")
-
-
-
-.. parsed-literal::
-
- data/kinetics_classnames.json: 0.00B [00:00, ?B/s]
-
-
-Let us download a sample video to run inference on, and take a look at
-the downloaded video.
-
-.. code:: ipython3
-
- VIDEO_SOURCE = "https://dl.fbaipublicfiles.com/pytorchvideo/projects/archery.mp4"
- VIDEO_NAME = "archery.mp4"
- VIDEO_PATH = DATA_DIR / VIDEO_NAME
-
- download_file(url=VIDEO_SOURCE, directory=DATA_DIR, show_progress=True)
- Video(VIDEO_PATH, embed=True)
-
-
-
-.. parsed-literal::
-
- data/archery.mp4: 0%| | 0.00/536k [00:00
-
- Your browser does not support the video tag.
-
-
-
-
-The sample video requires some preprocessing before we can run inference
-on it. During preprocessing, the video is normalized and scaled to size.
-Additionally, this preprocessing pipeline also involves sampling frames
-from the video to pass through the two pathways. The slow pathway can be
-any convolutional network that uses a large temporal stride on the input
-frames. The fast pathway is another convolutional network that uses a
-temporal stride smaller by a factor alpha(\ :math:`\alpha`). In our
-model, both pathways use a 3D ResNet model. We define the following
-helper functions to implement the preprocessing steps.
-
-.. code:: ipython3
-
- def scale_short_side(size: int, frame: np.ndarray) -> np.ndarray:
- """
- Scale the short side of the frame to size and return a float
- array.
- """
- height = frame.shape[0]
- width = frame.shape[1]
- # return unchanged if short side already scaled
- if (width <= height and width == size) or (height <= width and height == size):
- return frame
- new_width = size
- new_height = size
- if width < height:
- new_height = int(math.floor((float(height) / width) * size))
- else:
- new_width = int(math.floor((float(width) / height) * size))
- scaled = cv2.resize(frame, (new_width, new_height), interpolation=cv2.INTER_LINEAR)
- return scaled.astype(np.float32)
-
-
- def center_crop(size: int, frame: np.ndarray) -> np.ndarray:
- """
- Center crop the input frame to size.
- """
- height = frame.shape[0]
- width = frame.shape[1]
- y_offset = int(math.ceil((height - size) / 2))
- x_offset = int(math.ceil((width - size) / 2))
- cropped = frame[y_offset : y_offset + size, x_offset : x_offset + size, :]
- assert cropped.shape[0] == size, "Image height not cropped properly"
- assert cropped.shape[1] == size, "Image width not cropped properly"
- return cropped
-
-
- def normalize(array: np.ndarray, mean: List[float], std: List[float]) -> np.ndarray:
- """
- Normalize a given array by subtracting the mean and dividing the std.
- """
- if array.dtype == np.uint8:
- array = array.astype(np.float32)
- array = array / 255.0
- mean = np.array(mean, dtype=np.float32)
- std = np.array(std, dtype=np.float32)
- array = array - mean
- array = array / std
- return array
-
-
- def pack_pathway_output(frames: np.ndarray, alpha: int = 4) -> List[np.ndarray]:
- """
- Prepare output as a list of arrays, each corresponding
- to a unique pathway.
- """
- fast_pathway = frames
- # Perform temporal sampling from the fast pathway.
- slow_pathway = np.take(
- frames,
- indices=np.linspace(0, frames.shape[1] - 1, frames.shape[1] // alpha).astype(np.int_),
- axis=1,
- )
- frame_list = [slow_pathway, fast_pathway]
- return frame_list
-
-
- def process_inputs(
- frames: List[np.ndarray],
- num_frames: int,
- crop_size: int,
- mean: List[float],
- std: List[float],
- ) -> List[np.ndarray]:
- """
- Performs normalization and applies required transforms
- to prepare the input frames and returns a list of arrays.
- Specifically the following actions are performed
- 1. scale the short side of the frames
- 2. center crop the frames to crop_size
- 3. perform normalization by subtracting mean and dividing std
- 4. sample frames for specified num_frames
- 5. sample frames for slow and fast pathways
- """
- inputs = [scale_short_side(size=crop_size, frame=frame) for frame in frames]
- inputs = [center_crop(size=crop_size, frame=frame) for frame in inputs]
- inputs = np.array(inputs).astype(np.float32) / 255
- inputs = normalize(array=inputs, mean=mean, std=std)
- # T H W C -> C T H W
- inputs = inputs.transpose([3, 0, 1, 2])
- # Sample frames for num_frames specified
- indices = np.linspace(0, inputs.shape[1] - 1, num_frames).astype(np.int_)
- inputs = np.take(inputs, indices=indices, axis=1)
- # prepare pathways for the model
- inputs = pack_pathway_output(inputs)
- inputs = [np.expand_dims(inp, 0) for inp in inputs]
- return inputs
-
-Another helper method to run inference on a custom video using the given
-model.
-
-.. code:: ipython3
-
- def run_inference(
- model: Any,
- video_path: str,
- top_k: int,
- id_to_label_mapping: Dict[str, str],
- num_frames: int,
- sampling_rate: int,
- crop_size: int,
- mean: List[float],
- std: List[float],
- ) -> List[str]:
- """
- Run inference on the video given by video_path using the given model.
- First, the video is loaded from source. Frames are collected, processed
- and fed to the model. The top top_k predicted class IDs are converted to class
- labels and returned as a list of strings.
- """
- video_cap = cv2.VideoCapture(video_path)
- frames = []
- seq_length = num_frames * sampling_rate
- # get the list of frames from the video
- ret = True
- while ret and len(frames) < seq_length:
- ret, frame = video_cap.read()
- frames.append(frame)
- # prepare the inputs
- inputs = process_inputs(frames=frames, num_frames=num_frames, crop_size=crop_size, mean=mean, std=std)
-
- if isinstance(model, ov.CompiledModel):
- # openvino compiled model
- output_blob = model.output(0)
- predictions = model(inputs)[output_blob]
- else:
- # pytorch model
- predictions = model([torch.from_numpy(inp) for inp in inputs])
- predictions = predictions.detach().cpu().numpy()
-
- def softmax(x):
- return np.exp(x) / np.exp(x).sum(axis=None)
-
- # apply activation
- predictions = softmax(predictions)
-
- # top k predicted class IDs
- topk = 5
- pred_classes = np.argsort(-1 * predictions, axis=1)[:, :topk]
-
- # Map the predicted classes to the label names
- pred_class_names = [id_to_label_mapping[int(i)] for i in pred_classes[0]]
- return pred_class_names
-
-We define model-specific parameters for processing the input and run
-inference using the same. The top 5 predictions can be seen below.
-
-.. code:: ipython3
-
- NUM_FRAMES = 32
- SAMPLING_RATE = 2
- CROP_SIZE = 256
- MEAN = [0.45, 0.45, 0.45]
- STD = [0.225, 0.225, 0.225]
- TOP_K = 5
-
- predictions = run_inference(
- model=model,
- video_path=str(VIDEO_PATH),
- top_k=TOP_K,
- id_to_label_mapping=kinetics_id_to_classname,
- num_frames=NUM_FRAMES,
- sampling_rate=SAMPLING_RATE,
- crop_size=CROP_SIZE,
- mean=MEAN,
- std=STD,
- )
-
- print(f"Predicted labels: {', '.join(predictions)}")
-
-
-.. parsed-literal::
-
- Predicted labels: archery, throwing axe, playing paintball, golf driving, riding or walking with horse
-
-
-Convert model to OpenVINO Intermediate Representation
------------------------------------------------------
-
-
-
-Now that we have obtained our trained model and checked inference with
-it, we export the PyTorch model to OpenVINO IR format. In this format,
-the network is represented using two files: an ``xml`` file describing
-the network architecture and an accompanying binary file that stores
-constant values such as convolution weights in a binary format. We can
-use model conversion API for converting into IR format as follows. The
-``ov.convert_model`` method returns an ``ov.Model`` object that can
-either be compiled and inferred or serialized.
-
-.. code:: ipython3
-
- class ModelWrapper(torch.nn.Module):
- def __init__(self, model):
- super().__init__()
- self.model = model
-
- def forward(self, input):
- return model(list(input))
-
-
- dummy_input = [torch.randn((1, 3, 8, 256, 256)), torch.randn([1, 3, 32, 256, 256])]
-
- model = ov.convert_model(ModelWrapper(model), example_input=(dummy_input,))
- IR_PATH = MODEL_DIR / "slowfast-r50.xml"
-
-
- # serialize model for saving IR
- ov.save_model(model=model, output_model=str(IR_PATH), compress_to_fp16=False)
-
-Next, we read and compile the serialized model using OpenVINO runtime.
-The ``read_model`` function expects the ``.bin`` weights file to have
-the same filename and be located in the same directory as the ``.xml``
-file. If the weights file has a different filename, it can be specified
-using the ``weights`` parameter.
-
-.. code:: ipython3
-
- core = ov.Core()
-
- # read converted model
- conv_model = core.read_model(str(IR_PATH))
-
-Select inference device
------------------------
-
-
-
-select device from dropdown list for running inference using OpenVINO
-
-.. code:: ipython3
-
- import ipywidgets as widgets
-
- device = widgets.Dropdown(
- options=core.available_devices + ["AUTO"],
- value="AUTO",
- description="Device:",
- disabled=False,
- )
-
- device
-
-
-
-
-.. parsed-literal::
-
- Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
-
-
-
-.. code:: ipython3
-
- # load model on device
- compiled_model = core.compile_model(model=conv_model, device_name=device.value)
-
-Verify Model Inference
-----------------------
-
-
-
-Using the compiled model, we run inference on the same sample video and
-print the top 5 predictions again.
-
-.. code:: ipython3
-
- pred_class_names = run_inference(
- model=compiled_model,
- video_path=str(VIDEO_PATH),
- top_k=TOP_K,
- id_to_label_mapping=kinetics_id_to_classname,
- num_frames=NUM_FRAMES,
- sampling_rate=SAMPLING_RATE,
- crop_size=CROP_SIZE,
- mean=MEAN,
- std=STD,
- )
- print(f"Predicted labels: {', '.join(pred_class_names)}")
-
-
-.. parsed-literal::
-
- Predicted labels: archery, throwing axe, playing paintball, golf driving, riding or walking with horse
-
diff --git a/docs/notebooks/sparsity-optimization-with-output.rst b/docs/notebooks/sparsity-optimization-with-output.rst
index 0702349fa2f..206192ad9c5 100644
--- a/docs/notebooks/sparsity-optimization-with-output.rst
+++ b/docs/notebooks/sparsity-optimization-with-output.rst
@@ -51,20 +51,8 @@ Prerequisites
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -83,11 +71,6 @@ Imports
from huggingface_hub import hf_hub_download
-.. parsed-literal::
-
- OpenVINO Tokenizer version is not compatible with OpenVINO version. Installed OpenVINO version: 2024.0.0,OpenVINO Tokenizers requires . OpenVINO Tokenizers models will not be added during export.
-
-
.. parsed-literal::
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
@@ -95,19 +78,11 @@ Imports
.. parsed-literal::
- 2024-04-18 00:51:56.238501: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:51:56.274301: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 01:29:43.114992: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 01:29:43.150328: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:51:56.850457: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
+ 2024-05-07 01:29:43.748064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
@@ -144,6 +119,8 @@ model card on Hugging Face.
.. parsed-literal::
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Compiling the model to CPU ...
@@ -220,26 +197,18 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
-
-
-.. parsed-literal::
-
[ INFO ] CPU
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 61.16 ms
+ [ INFO ] Read model took 68.64 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@@ -250,11 +219,7 @@ as an example. It is recommended to tune based on your applications.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
-
-
-.. parsed-literal::
-
- [ INFO ] Reshape model took 22.79 ms
+ [ INFO ] Reshape model took 30.26 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@@ -263,11 +228,7 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Model outputs:
[ INFO ] logits (node: logits) : f32 / [...] / [1,2]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 1249.77 ms
+ [ INFO ] Compile model took 1217.01 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: torch_jit
@@ -282,6 +243,7 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
@@ -298,25 +260,17 @@ as an example. It is recommended to tune based on your applications.
[ INFO ] Fill input 'token_type_ids' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 28.22 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 25.94 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 9052 iterations
- [ INFO ] Duration: 60042.04 ms
+ [ INFO ] Count: 9352 iterations
+ [ INFO ] Duration: 60034.37 ms
[ INFO ] Latency:
- [ INFO ] Median: 26.38 ms
- [ INFO ] Average: 26.41 ms
- [ INFO ] Min: 24.29 ms
- [ INFO ] Max: 41.86 ms
- [ INFO ] Throughput: 150.76 FPS
+ [ INFO ] Median: 25.33 ms
+ [ INFO ] Average: 25.43 ms
+ [ INFO ] Min: 23.63 ms
+ [ INFO ] Max: 39.24 ms
+ [ INFO ] Throughput: 155.78 FPS
Benchmark quantized sparse inference performance
@@ -362,26 +316,18 @@ for which a layer will be enabled.
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
-
-
-.. parsed-literal::
-
[ INFO ] CPU
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 68.00 ms
+ [ INFO ] Read model took 65.15 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
@@ -392,11 +338,7 @@ for which a layer will be enabled.
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,64], 'attention_mask': [1,64], 'token_type_ids': [1,64]
-
-
-.. parsed-literal::
-
- [ INFO ] Reshape model took 23.17 ms
+ [ INFO ] Reshape model took 30.54 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,64]
@@ -407,18 +349,18 @@ for which a layer will be enabled.
[Step 7/11] Loading the model to the device
[ ERROR ] Exception from src/inference/src/cpp/core.cpp:106:
Exception from src/inference/src/dev/plugin.cpp:54:
- Exception from src/plugins/intel_cpu/src/config.cpp:208:
+ Exception from src/plugins/intel_cpu/src/config.cpp:226:
Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers
Traceback (most recent call last):
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 408, in main
+ File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 408, in main
compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config)
- File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/runtime/ie_api.py", line 515, in compile_model
+ File "/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/runtime/ie_api.py", line 521, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:106:
Exception from src/inference/src/dev/plugin.cpp:54:
- Exception from src/plugins/intel_cpu/src/config.cpp:208:
+ Exception from src/plugins/intel_cpu/src/config.cpp:226:
Wrong value for property key CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE. Expected only float numbers
diff --git a/docs/notebooks/speech-recognition-quantization-wav2vec2-with-output.rst b/docs/notebooks/speech-recognition-quantization-wav2vec2-with-output.rst
index 414f26f6f15..2e201041dc3 100644
--- a/docs/notebooks/speech-recognition-quantization-wav2vec2-with-output.rst
+++ b/docs/notebooks/speech-recognition-quantization-wav2vec2-with-output.rst
@@ -45,103 +45,53 @@ Table of contents:
.. parsed-literal::
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
- Requirement already satisfied: datasets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.18.0)
- Requirement already satisfied: torchmetrics>=0.11.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (1.3.2)
- Requirement already satisfied: torch>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.2.2+cpu)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.13.4)
- Requirement already satisfied: numpy>=1.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (1.23.5)
- Requirement already satisfied: pyarrow>=12.0.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (15.0.2)
- Requirement already satisfied: pyarrow-hotfix in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.6)
- Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.3.8)
- Requirement already satisfied: pandas in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (2.0.3)
- Requirement already satisfied: requests>=2.19.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (2.31.0)
- Requirement already satisfied: tqdm>=4.62.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (4.66.2)
- Requirement already satisfied: xxhash in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.4.1)
- Requirement already satisfied: multiprocess in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.70.16)
- Requirement already satisfied: fsspec<=2024.2.0,>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from fsspec[http]<=2024.2.0,>=2023.1.0->datasets) (2024.2.0)
- Requirement already satisfied: aiohttp in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.9.5)
- Requirement already satisfied: huggingface-hub>=0.19.4 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.22.2)
- Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (24.0)
- Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (6.0.1)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torchmetrics>=0.11.0) (0.11.2)
- Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torchmetrics>=0.11.0) (4.11.0)
- Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (1.12)
- Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (3.1)
- Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (3.1.3)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: aiosignal>=1.1.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.3.1)
- Requirement already satisfied: attrs>=17.3.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (23.2.0)
- Requirement already satisfied: frozenlist>=1.1.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.4.1)
- Requirement already satisfied: multidict<7.0,>=4.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (6.0.5)
- Requirement already satisfied: yarl<2.0,>=1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.9.4)
- Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (4.0.3)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: setuptools in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from lightning-utilities>=0.8.0->torchmetrics>=0.11.0) (69.5.1)
- Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.3.2)
- Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.7)
- Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2.2.1)
- Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2024.2.2)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1.0) (2.1.5)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: python-dateutil>=2.8.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2.9.0.post0)
- Requirement already satisfied: pytz>=2020.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2024.1)
- Requirement already satisfied: tzdata>=2022.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2024.1)
- Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1.0) (1.3.0)
-
-
-.. parsed-literal::
-
- Requirement already satisfied: six>=1.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
-
-
-.. parsed-literal::
-
+ Requirement already satisfied: datasets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.19.1)
+ Requirement already satisfied: torchmetrics>=0.11.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (1.4.0)
+ Requirement already satisfied: torch>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.3.0+cpu)
+ Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.14.0)
+ Requirement already satisfied: numpy>=1.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (1.23.5)
+ Requirement already satisfied: pyarrow>=12.0.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (16.0.0)
+ Requirement already satisfied: pyarrow-hotfix in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.6)
+ Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.3.8)
+ Requirement already satisfied: pandas in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (2.0.3)
+ Requirement already satisfied: requests>=2.19.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (2.31.0)
+ Requirement already satisfied: tqdm>=4.62.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (4.66.4)
+ Requirement already satisfied: xxhash in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.4.1)
+ Requirement already satisfied: multiprocess in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.70.16)
+ Requirement already satisfied: fsspec<=2024.3.1,>=2023.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from fsspec[http]<=2024.3.1,>=2023.1.0->datasets) (2024.3.1)
+ Requirement already satisfied: aiohttp in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (3.9.5)
+ Requirement already satisfied: huggingface-hub>=0.21.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (0.23.0)
+ Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (24.0)
+ Requirement already satisfied: pyyaml>=5.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from datasets) (6.0.1)
+ Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torchmetrics>=0.11.0) (0.11.2)
+ Requirement already satisfied: pretty-errors==1.2.25 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torchmetrics>=0.11.0) (1.2.25)
+ Requirement already satisfied: typing-extensions in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torchmetrics>=0.11.0) (4.11.0)
+ Requirement already satisfied: colorama in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pretty-errors==1.2.25->torchmetrics>=0.11.0) (0.4.6)
+ Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (1.12)
+ Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (3.1)
+ Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1.0) (3.1.4)
+ Requirement already satisfied: aiosignal>=1.1.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.3.1)
+ Requirement already satisfied: attrs>=17.3.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (23.2.0)
+ Requirement already satisfied: frozenlist>=1.1.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.4.1)
+ Requirement already satisfied: multidict<7.0,>=4.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (6.0.5)
+ Requirement already satisfied: yarl<2.0,>=1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (1.9.4)
+ Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from aiohttp->datasets) (4.0.3)
+ Requirement already satisfied: setuptools in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from lightning-utilities>=0.8.0->torchmetrics>=0.11.0) (69.5.1)
+ Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.3.2)
+ Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.7)
+ Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2.2.1)
+ Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2024.2.2)
+ Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1.0) (2.1.5)
+ Requirement already satisfied: python-dateutil>=2.8.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2.9.0.post0)
+ Requirement already satisfied: pytz>=2020.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2024.1)
+ Requirement already satisfied: tzdata>=2022.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas->datasets) (2024.1)
+ Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1.0) (1.3.0)
+ Requirement already satisfied: six>=1.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
-
-
-.. parsed-literal::
-
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
-
-
-.. parsed-literal::
-
Note: you may need to restart the kernel to use updated packages.
@@ -190,13 +140,11 @@ IR).
.. parsed-literal::
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
+ warnings.warn(
Some weights of the model checkpoint at facebook/wav2vec2-base-960h were not used when initializing Wav2Vec2ForCTC: ['wav2vec2.encoder.pos_conv_embed.conv.weight_g', 'wav2vec2.encoder.pos_conv_embed.conv.weight_v']
- This IS expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
-
-
-.. parsed-literal::
-
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-base-960h and are newly initialized: ['wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original1', 'wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
@@ -213,15 +161,11 @@ IR).
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
warnings.warn(
-
-
-.. parsed-literal::
-
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:594: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:588: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:633: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:627: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
@@ -264,7 +208,7 @@ dataset.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1461: FutureWarning: The repository for patrickvonplaten/librispeech_asr_dummy contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/patrickvonplaten/librispeech_asr_dummy
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/datasets/load.py:1486: FutureWarning: The repository for patrickvonplaten/librispeech_asr_dummy contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/patrickvonplaten/librispeech_asr_dummy
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
warnings.warn(
@@ -339,14 +283,10 @@ steps:
.. parsed-literal::
- 2024-04-18 00:53:31.426814: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
- 2024-04-18 00:53:31.462819: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+ 2024-05-07 01:31:19.450719: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
+ 2024-05-07 01:31:19.486606: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-
-
-.. parsed-literal::
-
- 2024-04-18 00:53:32.048658: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+ 2024-05-07 01:31:20.050394: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
@@ -394,36 +334,12 @@ steps:
.. parsed-literal::
INFO:nncf:4 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:36 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
INFO:nncf:50 ignored nodes were found by name in the NNCFGraph
-
-
-.. parsed-literal::
-
- INFO:nncf:Not adding activation input quantizer for operation: 3 __module.wav2vec2.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution
-
-
-.. parsed-literal::
-
- INFO:nncf:Not adding activation input quantizer for operation: 11 __module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution
-
-
-.. parsed-literal::
-
- INFO:nncf:Not adding activation input quantizer for operation: 13 __module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution
-
-
-.. parsed-literal::
-
- INFO:nncf:Not adding activation input quantizer for operation: 15 __module.wav2vec2.feature_extractor.conv_layers.3.conv/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 2 __module.wav2vec2.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 10 __module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 12 __module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution
+ INFO:nncf:Not adding activation input quantizer for operation: 14 __module.wav2vec2.feature_extractor.conv_layers.3.conv/aten::_convolution/Convolution
@@ -624,7 +540,7 @@ quantized model.
.. parsed-literal::
- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-661/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:62: FutureWarning: Importing `WordErrorRate` from `torchmetrics` was deprecated and will be removed in 2.0. Import `WordErrorRate` from `torchmetrics.text` instead.
+ /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:62: FutureWarning: Importing `WordErrorRate` from `torchmetrics` was deprecated and will be removed in 2.0. Import `WordErrorRate` from `torchmetrics.text` instead.
_future_warning(
@@ -683,42 +599,34 @@ models.
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
- [ INFO ] Read model took 27.44 ms
+ [ INFO ] Read model took 26.32 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
- [ INFO ] input_values (node: input_values) : f32 / [...] / [?,?]
+ [ INFO ] input_values , 46 (node: input_values) : f32 / [...] / [?,?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
- [ INFO ] Reshaping model: 'input_values': [1,30480]
-
-
-.. parsed-literal::
-
- [ INFO ] Reshape model took 30.09 ms
+ [ INFO ] Reshaping model: '46': [1,30480]
+ [ INFO ] Reshape model took 63.78 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
- [ INFO ] input_values (node: input_values) : f32 / [...] / [1,30480]
+ [ INFO ] input_values , 46 (node: input_values) : f32 / [...] / [1,30480]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 524.97 ms
+ [ INFO ] Compile model took 523.09 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: Model0
@@ -739,6 +647,7 @@ models.
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@@ -748,30 +657,23 @@ models.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
- [ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!
- [ INFO ] Fill input 'input_values' with random values
+ [ WARNING ] No input files were given for input '46'!. This input will be filled with random values!
+ [ INFO ] Fill input '46' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 57.09 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 62.11 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 5502 iterations
- [ INFO ] Duration: 120210.49 ms
+ [ INFO ] Count: 5538 iterations
+ [ INFO ] Duration: 120109.43 ms
[ INFO ] Latency:
- [ INFO ] Median: 130.91 ms
- [ INFO ] Average: 130.93 ms
- [ INFO ] Min: 69.78 ms
- [ INFO ] Max: 154.76 ms
- [ INFO ] Throughput: 45.77 FPS
+ [ INFO ] Median: 129.95 ms
+ [ INFO ] Average: 129.96 ms
+ [ INFO ] Min: 66.70 ms
+ [ INFO ] Max: 153.18 ms
+ [ INFO ] Throughput: 46.11 FPS
.. code:: ipython3
@@ -787,52 +689,36 @@ models.
[Step 2/11] Loading OpenVINO Runtime
[ WARNING ] Default duration 120 seconds is used for unknown device AUTO
[ INFO ] OpenVINO:
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] AUTO
- [ INFO ] Build ................................. 2024.0.0-14509-34caeefd078-releases/2024/0
+ [ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
-
-
-.. parsed-literal::
-
- [ INFO ] Read model took 41.61 ms
+ [ INFO ] Read model took 32.12 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
- [ INFO ] input_values (node: input_values) : f32 / [...] / [?,?]
+ [ INFO ] 46 , input_values (node: input_values) : f32 / [...] / [?,?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
- [ INFO ] Reshaping model: 'input_values': [1,30480]
-
-
-.. parsed-literal::
-
- [ INFO ] Reshape model took 35.71 ms
+ [ INFO ] Reshaping model: '46': [1,30480]
+ [ INFO ] Reshape model took 65.21 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
- [ INFO ] input_values (node: input_values) : f32 / [...] / [1,30480]
+ [ INFO ] 46 , input_values (node: input_values) : f32 / [...] / [1,30480]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]
[Step 7/11] Loading the model to the device
-
-
-.. parsed-literal::
-
- [ INFO ] Compile model took 1265.41 ms
+ [ INFO ] Compile model took 1232.84 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
-
-
-.. parsed-literal::
-
[ INFO ] NETWORK_NAME: Model0
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
@@ -851,6 +737,7 @@ models.
[ INFO ] INFERENCE_PRECISION_HINT:
[ INFO ] KV_CACHE_PRECISION:
[ INFO ] LOG_LEVEL: Level.NO
+ [ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] NETWORK_NAME: Model0
[ INFO ] NUM_STREAMS: 6
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
@@ -860,28 +747,21 @@ models.
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_PRIORITY: Priority.MEDIUM
[ INFO ] LOADED_FROM_CACHE: False
+ [ INFO ] PERF_COUNT: False
[Step 9/11] Creating infer requests and preparing input tensors
- [ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!
- [ INFO ] Fill input 'input_values' with random values
+ [ WARNING ] No input files were given for input '46'!. This input will be filled with random values!
+ [ INFO ] Fill input '46' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 120000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
-
-
-.. parsed-literal::
-
- [ INFO ] First inference took 55.32 ms
-
-
-.. parsed-literal::
-
+ [ INFO ] First inference took 56.24 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
- [ INFO ] Count: 8274 iterations
- [ INFO ] Duration: 120148.65 ms
+ [ INFO ] Count: 8340 iterations
+ [ INFO ] Duration: 120110.08 ms
[ INFO ] Latency:
- [ INFO ] Median: 86.99 ms
- [ INFO ] Average: 86.98 ms
- [ INFO ] Min: 44.88 ms
- [ INFO ] Max: 112.37 ms
- [ INFO ] Throughput: 68.86 FPS
+ [ INFO ] Median: 86.26 ms
+ [ INFO ] Average: 86.26 ms
+ [ INFO ] Min: 39.87 ms
+ [ INFO ] Max: 110.38 ms
+ [ INFO ] Throughput: 69.44 FPS
diff --git a/docs/notebooks/ssd-b1-with-output.rst b/docs/notebooks/ssd-b1-with-output.rst
deleted file mode 100644
index d7cc34be426..00000000000
--- a/docs/notebooks/ssd-b1-with-output.rst
+++ /dev/null
@@ -1,787 +0,0 @@
-Image generation with Segmind Stable Diffusion 1B (SSD-1B) model and OpenVINO
-=============================================================================
-
-The `Segmind Stable Diffusion Model
-(SSD-1B)