995 lines
35 KiB
ReStructuredText
995 lines
35 KiB
ReStructuredText
Visual Content Search using MobileCLIP and OpenVINO
|
||
===================================================
|
||
|
||
Semantic visual content search is a machine learning task that uses
|
||
either a text query or an input image to search a database of images
|
||
(photo gallery, video) to find images that are semantically similar to
|
||
the search query. Historically, building a robust search engine for
|
||
images was difficult. One could search by features such as file name and
|
||
image metadata, and use any context around an image (i.e. alt text or
|
||
surrounding text if an image appears in a passage of text) to provide
|
||
the richer searching feature. This was before the advent of neural
|
||
networks that can identify semantically related images to a given user
|
||
query.
|
||
|
||
`Contrastive Language-Image Pre-Training
|
||
(CLIP) <https://arxiv.org/abs/2103.00020>`__ models provide the means
|
||
through which you can implement a semantic search engine with a few
|
||
dozen lines of code. The CLIP model has been trained on millions of
|
||
pairs of text and images, encoding semantics from images and text
|
||
combined. Using CLIP, you can provide a text query and CLIP will return
|
||
the images most related to the query.
|
||
|
||
In this tutorial, we consider how to use MobileCLIP to implement a
|
||
visual content search engine for finding relevant frames in video. ####
|
||
Table of contents:
|
||
|
||
- `Prerequisites <#prerequisites>`__
|
||
- `Select model <#select-model>`__
|
||
- `Run model inference <#run-model-inference>`__
|
||
|
||
- `Prepare image gallery <#prepare-image-gallery>`__
|
||
- `Prepare model <#prepare-model>`__
|
||
- `Perform search <#perform-search>`__
|
||
|
||
- `Convert Model to OpenVINO Intermediate Representation
|
||
format <#convert-model-to-openvino-intermediate-representation-format>`__
|
||
- `Run OpenVINO model inference <#run-openvino-model-inference>`__
|
||
|
||
- `Select device for image
|
||
encoder <#select-device-for-image-encoder>`__
|
||
- `Select device for text
|
||
encoder <#select-device-for-text-encoder>`__
|
||
- `Perform search <#perform-search>`__
|
||
|
||
- `Interactive Demo <#interactive-demo>`__
|
||
|
||
Prerequisites
|
||
-------------
|
||
|
||
## Prerequisites
|
||
|
||
.. code:: ipython3
|
||
|
||
from pathlib import Path
|
||
|
||
repo_dir = Path("./ml-mobileclip")
|
||
|
||
if not repo_dir.exists():
|
||
!git clone https://github.com/apple/ml-mobileclip.git
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Cloning into 'ml-mobileclip'...
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
remote: Enumerating objects: 45, done.[K
|
||
remote: Counting objects: 2% (1/45)[K
|
||
remote: Counting objects: 4% (2/45)[K
|
||
remote: Counting objects: 6% (3/45)[K
|
||
remote: Counting objects: 8% (4/45)[K
|
||
remote: Counting objects: 11% (5/45)[K
|
||
remote: Counting objects: 13% (6/45)[K
|
||
remote: Counting objects: 15% (7/45)[K
|
||
remote: Counting objects: 17% (8/45)[K
|
||
remote: Counting objects: 20% (9/45)[K
|
||
remote: Counting objects: 22% (10/45)[K
|
||
remote: Counting objects: 24% (11/45)[K
|
||
remote: Counting objects: 26% (12/45)[K
|
||
remote: Counting objects: 28% (13/45)[K
|
||
remote: Counting objects: 31% (14/45)[K
|
||
remote: Counting objects: 33% (15/45)[K
|
||
remote: Counting objects: 35% (16/45)[K
|
||
remote: Counting objects: 37% (17/45)[K
|
||
remote: Counting objects: 40% (18/45)[K
|
||
remote: Counting objects: 42% (19/45)[K
|
||
remote: Counting objects: 44% (20/45)[K
|
||
remote: Counting objects: 46% (21/45)[K
|
||
remote: Counting objects: 48% (22/45)[K
|
||
remote: Counting objects: 51% (23/45)[K
|
||
remote: Counting objects: 53% (24/45)[K
|
||
remote: Counting objects: 55% (25/45)[K
|
||
remote: Counting objects: 57% (26/45)[K
|
||
remote: Counting objects: 60% (27/45)[K
|
||
remote: Counting objects: 62% (28/45)[K
|
||
remote: Counting objects: 64% (29/45)[K
|
||
remote: Counting objects: 66% (30/45)[K
|
||
remote: Counting objects: 68% (31/45)[K
|
||
remote: Counting objects: 71% (32/45)[K
|
||
remote: Counting objects: 73% (33/45)[K
|
||
remote: Counting objects: 75% (34/45)[K
|
||
remote: Counting objects: 77% (35/45)[K
|
||
remote: Counting objects: 80% (36/45)[K
|
||
remote: Counting objects: 82% (37/45)[K
|
||
remote: Counting objects: 84% (38/45)[K
|
||
remote: Counting objects: 86% (39/45)[K
|
||
remote: Counting objects: 88% (40/45)[K
|
||
remote: Counting objects: 91% (41/45)[K
|
||
remote: Counting objects: 93% (42/45)[K
|
||
remote: Counting objects: 95% (43/45)[K
|
||
remote: Counting objects: 97% (44/45)[K
|
||
remote: Counting objects: 100% (45/45)[K
|
||
remote: Counting objects: 100% (45/45), done.[K
|
||
remote: Compressing objects: 2% (1/36)[K
|
||
remote: Compressing objects: 5% (2/36)[K
|
||
remote: Compressing objects: 8% (3/36)[K
|
||
remote: Compressing objects: 11% (4/36)[K
|
||
remote: Compressing objects: 13% (5/36)[K
|
||
remote: Compressing objects: 16% (6/36)[K
|
||
remote: Compressing objects: 19% (7/36)[K
|
||
remote: Compressing objects: 22% (8/36)[K
|
||
remote: Compressing objects: 25% (9/36)[K
|
||
remote: Compressing objects: 27% (10/36)[K
|
||
remote: Compressing objects: 30% (11/36)[K
|
||
remote: Compressing objects: 33% (12/36)[K
|
||
remote: Compressing objects: 36% (13/36)[K
|
||
remote: Compressing objects: 38% (14/36)[K
|
||
remote: Compressing objects: 41% (15/36)[K
|
||
remote: Compressing objects: 44% (16/36)[K
|
||
remote: Compressing objects: 47% (17/36)[K
|
||
remote: Compressing objects: 50% (18/36)[K
|
||
remote: Compressing objects: 52% (19/36)[K
|
||
remote: Compressing objects: 55% (20/36)[K
|
||
remote: Compressing objects: 58% (21/36)[K
|
||
remote: Compressing objects: 61% (22/36)[K
|
||
remote: Compressing objects: 63% (23/36)[K
|
||
remote: Compressing objects: 66% (24/36)[K
|
||
remote: Compressing objects: 69% (25/36)[K
|
||
remote: Compressing objects: 72% (26/36)[K
|
||
remote: Compressing objects: 75% (27/36)[K
|
||
remote: Compressing objects: 77% (28/36)[K
|
||
remote: Compressing objects: 80% (29/36)[K
|
||
remote: Compressing objects: 83% (30/36)[K
|
||
remote: Compressing objects: 86% (31/36)[K
|
||
remote: Compressing objects: 88% (32/36)[K
|
||
remote: Compressing objects: 91% (33/36)[K
|
||
remote: Compressing objects: 94% (34/36)[K
|
||
remote: Compressing objects: 97% (35/36)[K
|
||
remote: Compressing objects: 100% (36/36)[K
|
||
remote: Compressing objects: 100% (36/36), done.[K
|
||
Unpacking objects: 2% (1/45)
|
||
Unpacking objects: 4% (2/45)
|
||
Unpacking objects: 6% (3/45)
|
||
Unpacking objects: 8% (4/45)
|
||
Unpacking objects: 11% (5/45)
|
||
|
||
.. parsed-literal::
|
||
|
||
Unpacking objects: 13% (6/45)
|
||
Unpacking objects: 15% (7/45)
|
||
Unpacking objects: 17% (8/45)
|
||
Unpacking objects: 20% (9/45)
|
||
Unpacking objects: 22% (10/45)
|
||
Unpacking objects: 24% (11/45)
|
||
Unpacking objects: 26% (12/45)
|
||
|
||
.. parsed-literal::
|
||
|
||
remote: Total 45 (delta 9), reused 44 (delta 8), pack-reused 0[K
|
||
Unpacking objects: 28% (13/45)
|
||
Unpacking objects: 31% (14/45)
|
||
Unpacking objects: 33% (15/45)
|
||
Unpacking objects: 35% (16/45)
|
||
Unpacking objects: 37% (17/45)
|
||
Unpacking objects: 40% (18/45)
|
||
Unpacking objects: 42% (19/45)
|
||
Unpacking objects: 44% (20/45)
|
||
Unpacking objects: 46% (21/45)
|
||
Unpacking objects: 48% (22/45)
|
||
Unpacking objects: 51% (23/45)
|
||
Unpacking objects: 53% (24/45)
|
||
Unpacking objects: 55% (25/45)
|
||
Unpacking objects: 57% (26/45)
|
||
Unpacking objects: 60% (27/45)
|
||
Unpacking objects: 62% (28/45)
|
||
Unpacking objects: 64% (29/45)
|
||
Unpacking objects: 66% (30/45)
|
||
Unpacking objects: 68% (31/45)
|
||
Unpacking objects: 71% (32/45)
|
||
Unpacking objects: 73% (33/45)
|
||
Unpacking objects: 75% (34/45)
|
||
Unpacking objects: 77% (35/45)
|
||
Unpacking objects: 80% (36/45)
|
||
Unpacking objects: 82% (37/45)
|
||
Unpacking objects: 84% (38/45)
|
||
Unpacking objects: 86% (39/45)
|
||
Unpacking objects: 88% (40/45)
|
||
Unpacking objects: 91% (41/45)
|
||
Unpacking objects: 93% (42/45)
|
||
Unpacking objects: 95% (43/45)
|
||
Unpacking objects: 97% (44/45)
|
||
Unpacking objects: 100% (45/45)
|
||
Unpacking objects: 100% (45/45), 428.50 KiB | 2.86 MiB/s, done.
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
%pip install -q "./ml-mobileclip" --extra-index-url "https://download.pytorch.org/whl/cpu"
|
||
%pip install -q "openvino>=2024.0.0" "gradio" "matplotlib" "Pillow" "altair" "pandas" "opencv-python"
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
|
||
pyannote-audio 2.0.1 requires torchaudio<1.0,>=0.10, but you have torchaudio 2.2.1+cpu which is incompatible.
|
||
pytorch-lightning 1.6.5 requires protobuf<=3.20.1, but you have protobuf 4.25.3 which is incompatible.
|
||
torchaudio 2.2.1+cpu requires torch==2.2.1, but you have torch 1.13.1+cpu which is incompatible.
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
DEPRECATION: pytorch-lightning 1.6.5 has a non-standard dependency specifier torch>=1.8.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Note: you may need to restart the kernel to use updated packages.
|
||
|
||
|
||
Select model
|
||
------------
|
||
|
||
|
||
|
||
For starting work, we should select model that will be used in our
|
||
demonstration. By default, we will use the MobileCLIP model, but for
|
||
comparison purposes, you can select different models among:
|
||
|
||
- **CLIP** - CLIP (Contrastive Language-Image Pre-Training) is a neural
|
||
network trained on various (image, text) pairs. It can be instructed
|
||
in natural language to predict the most relevant text snippet, given
|
||
an image, without directly optimizing for the task. CLIP uses a
|
||
`ViT <https://arxiv.org/abs/2010.11929>`__ like transformer to get
|
||
visual features and a causal language model to get the text features.
|
||
The text and visual features are then projected into a latent space
|
||
with identical dimensions. The dot product between the projected
|
||
image and text features is then used as a similarity score. You can
|
||
find more information about this model in the `research
|
||
paper <https://arxiv.org/abs/2103.00020>`__, `OpenAI
|
||
blog <https://openai.com/blog/clip/>`__, `model
|
||
card <https://github.com/openai/CLIP/blob/main/model-card.md>`__ and
|
||
GitHub `repository <https://github.com/openai/CLIP>`__.
|
||
- **SigLIP** - The SigLIP model was proposed in `Sigmoid Loss for
|
||
Language Image Pre-Training <https://arxiv.org/abs/2303.15343>`__.
|
||
SigLIP proposes to replace the loss function used in
|
||
`CLIP <https://github.com/openai/CLIP>`__ (Contrastive Language–Image
|
||
Pre-training) by a simple pairwise sigmoid loss. This results in
|
||
better performance in terms of zero-shot classification accuracy on
|
||
ImageNet. You can find more information about this model in the
|
||
`research paper <https://arxiv.org/abs/2303.15343>`__ and `GitHub
|
||
repository <https://github.com/google-research/big_vision>`__,
|
||
- **MobileCLIP** - MobileCLIP – a new family of efficient image-text
|
||
models optimized for runtime performance along with a novel and
|
||
efficient training approach, namely multi-modal reinforced training.
|
||
The smallest variant MobileCLIP-S0 obtains similar zero-shot
|
||
performance as OpenAI’s CLIP ViT-b16 model while being several times
|
||
faster and 2.8x smaller. More details about model can be found in
|
||
`research paper <https://arxiv.org/pdf/2311.17049.pdf>`__ and `GitHub
|
||
repository <https://github.com/apple/ml-mobileclip>`__.
|
||
|
||
.. code:: ipython3
|
||
|
||
import ipywidgets as widgets
|
||
|
||
model_dir = Path("checkpoints")
|
||
|
||
supported_models = {
|
||
"MobileCLIP": {
|
||
"mobileclip_s0": {"model_name": "mobileclip_s0", "pretrained": model_dir / "mobileclip_s0.pt", "url": "https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s0.pt", "image_size": 256},
|
||
"mobileclip_s1": {"model_name": "mobileclip_s1", "pretrained": model_dir / "mobileclip_s1.pt", "url": "https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s1.pt", "image_size": 256},
|
||
"mobileclip_s2": {"model_name": "mobileclip_s0", "pretrained": model_dir / "mobileclip_s2.pt", "url": "https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s2.pt", "image_size": 256},
|
||
"mobileclip_b": {"model_name": "mobileclip_b", "pretrained": model_dir / "mobileclip_b.pt", "url": "https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_b.pt", "image_size": 224},
|
||
"mobileclip_blt": {"model_name": "mobileclip_b", "pretrained": model_dir / "mobileclip_blt.pt", "url": "https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_blt.pt", "image_size": 224},
|
||
},
|
||
"CLIP": {
|
||
"clip-vit-b-32": {"model_name": "ViT-B-32", "pretrained": "laion2b_s34b_b79k", "image_size": 224},
|
||
"clip-vit-b-16": {"image_name": "ViT-B-16", "pretrained": "openai", "image_size": 224},
|
||
"clip-vit-l-14": {"image_name": "ViT-L-14", "pretrained": "datacomp_xl_s13b_b90k", "image_size": 224},
|
||
"clip-vit-h-14": {"image_name": "ViT-H-14", "pretrained": "laion2b_s32b_b79k", "image_size": 224},
|
||
},
|
||
"SigLIP": {
|
||
"siglip-vit-b-16": {"model_name": "ViT-B-16-SigLIP", "pretrained": "webli", "image_size": 224},
|
||
"siglip-vit-l-16": {"model_name": "ViT-L-16-SigLIP-256", "pretrained": "webli", "image_size": 256},
|
||
}
|
||
|
||
}
|
||
|
||
|
||
model_type = widgets.Dropdown(options=supported_models.keys(), default="MobileCLIP", description="Model type:")
|
||
model_type
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Model type:', options=('MobileCLIP', 'CLIP', 'SigLIP'), value='MobileCLIP')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
available_models = supported_models[model_type.value]
|
||
|
||
model_checkpoint = widgets.Dropdown(options=available_models.keys(), default=list(available_models), description="Model:")
|
||
|
||
model_checkpoint
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Model:', options=('mobileclip_s0', 'mobileclip_s1', 'mobileclip_s2', 'mobileclip_b', 'mo…
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
import urllib.request
|
||
urllib.request.urlretrieve(
|
||
url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py',
|
||
filename='notebook_utils.py'
|
||
)
|
||
|
||
from notebook_utils import download_file
|
||
|
||
model_config = available_models[model_checkpoint.value]
|
||
|
||
Run model inference
|
||
-------------------
|
||
|
||
|
||
|
||
Now, let’s see model in action. We will try to find image, where some
|
||
specific object is represented using embeddings. Embeddings are a
|
||
numeric representation of data such as text and images. The model
|
||
learned to encode semantics about the contents of images in embedding
|
||
format. This ability turns the model into a powerful for solving various
|
||
tasks including image-text retrieval. To reach our goal we should:
|
||
|
||
1. Calculate embeddings for all of the images in our dataset;
|
||
2. Calculate a text embedding for a user query (i.e. “black dog” or
|
||
“car”);
|
||
3. Compare the text embedding to the image embeddings to find related
|
||
embeddings.
|
||
|
||
The closer two embeddings are, the more similar the contents they
|
||
represent are.
|
||
|
||
Prepare image gallery
|
||
~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
from typing import List
|
||
import matplotlib.pyplot as plt
|
||
import numpy as np
|
||
from PIL import Image
|
||
|
||
|
||
def visualize_result(images:List, query:str = "", selected:List[int] = None):
|
||
"""
|
||
Utility function for visualization classification results
|
||
params:
|
||
images (List[Image]) - list of images for visualization
|
||
query (str) - title for visualization
|
||
selected (List[int]) - list of selected image indices from images
|
||
returns:
|
||
matplotlib.Figure
|
||
"""
|
||
figsize = (20, 5)
|
||
fig, axs = plt.subplots(1, 4, figsize=figsize, sharex='all', sharey='all')
|
||
fig.patch.set_facecolor('white')
|
||
list_axes = list(axs.flat)
|
||
if query:
|
||
fig.suptitle(query, fontsize=20)
|
||
for idx, a in enumerate(list_axes):
|
||
a.set_xticklabels([])
|
||
a.set_yticklabels([])
|
||
a.get_xaxis().set_visible(False)
|
||
a.get_yaxis().set_visible(False)
|
||
a.grid(False)
|
||
a.imshow(images[idx])
|
||
if selected is not None and idx not in selected:
|
||
mask = np.ones_like(np.array(images[idx]))
|
||
a.imshow(mask, 'jet', interpolation='none', alpha=0.75)
|
||
return fig
|
||
|
||
|
||
images_urls = [
|
||
"https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/282ce53e-912d-41aa-ab48-2a001c022d74",
|
||
"https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/9bb40168-82b5-4b11-ada6-d8df104c736c",
|
||
"https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/0747b6db-12c3-4252-9a6a-057dcf8f3d4e",
|
||
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bricks.png"
|
||
]
|
||
image_names = ["red_panda.png", "cat.png", "raccoon.png", "dog.png"]
|
||
sample_path = Path("data")
|
||
sample_path.mkdir(parents=True, exist_ok=True)
|
||
|
||
images = []
|
||
for image_name, image_url in zip(image_names, images_urls):
|
||
image_path = sample_path / image_name
|
||
if not image_path.exists():
|
||
download_file(image_url, filename=image_name, directory=sample_path)
|
||
images.append(Image.open(image_path).convert("RGB").resize((640, 420)))
|
||
|
||
input_labels = ['cat']
|
||
text_descriptions = [f"This is a photo of a {label}" for label in input_labels]
|
||
|
||
visualize_result(images, "image gallery");
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/red_panda.png: 0%| | 0.00/50.6k [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/cat.png: 0%| | 0.00/54.5k [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/raccoon.png: 0%| | 0.00/106k [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/dog.png: 0%| | 0.00/716k [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. image:: 289-mobileclip-video-search-with-output_files/289-mobileclip-video-search-with-output_10_4.png
|
||
|
||
|
||
Prepare model
|
||
~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
The code bellow download model weights, create model class instance and
|
||
preprocessing utilities
|
||
|
||
.. code:: ipython3
|
||
|
||
import torch
|
||
import time
|
||
from PIL import Image
|
||
import mobileclip
|
||
import open_clip
|
||
|
||
# instantiate model
|
||
model_name = model_config["model_name"]
|
||
pretrained = model_config["pretrained"]
|
||
if model_type.value == "MobileCLIP":
|
||
model_dir.mkdir(exist_ok=True)
|
||
model_url = model_config["url"]
|
||
download_file(model_url, directory=model_dir)
|
||
model, _, preprocess = mobileclip.create_model_and_transforms(model_name, pretrained=pretrained)
|
||
tokenizer = mobileclip.get_tokenizer(model_name)
|
||
else:
|
||
model, _, preprocess = open_clip.create_model_and_transforms(model_name, pretrained=pretrained)
|
||
tokenizer = open_clip.get_tokenizer(model_name)
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
checkpoints/mobileclip_s0.pt: 0%| | 0.00/206M [00:00<?, ?B/s]
|
||
|
||
|
||
Perform search
|
||
~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
image_tensor = torch.stack([preprocess(image) for image in images])
|
||
text = tokenizer(text_descriptions)
|
||
|
||
|
||
with torch.no_grad():
|
||
|
||
# calculate image embeddings
|
||
image_encoding_start = time.perf_counter()
|
||
image_features = model.encode_image(image_tensor)
|
||
image_encoding_end = time.perf_counter()
|
||
print(f"Image encoding took {image_encoding_end - image_encoding_start:.3} ms")
|
||
# calculate text embeddings
|
||
text_encoding_start = time.perf_counter()
|
||
text_features = model.encode_text(text)
|
||
text_encoding_end = time.perf_counter()
|
||
print(f"Text encoding took {text_encoding_end - text_encoding_start:.3} ms")
|
||
|
||
# normalize embeddings
|
||
image_features /= image_features.norm(dim=-1, keepdim=True)
|
||
text_features /= text_features.norm(dim=-1, keepdim=True)
|
||
|
||
# calcualte similarity score
|
||
image_probs = (100.0 * text_features @ image_features.T).softmax(dim=-1)
|
||
selected_image = [torch.argmax(image_probs).item()]
|
||
|
||
visualize_result(images, input_labels[0], selected_image);
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Image encoding took 0.101 ms
|
||
Text encoding took 0.0102 ms
|
||
|
||
|
||
|
||
.. image:: 289-mobileclip-video-search-with-output_files/289-mobileclip-video-search-with-output_14_1.png
|
||
|
||
|
||
Convert Model to OpenVINO Intermediate Representation format
|
||
------------------------------------------------------------
|
||
|
||
|
||
|
||
For best results with OpenVINO, it is recommended to convert the model
|
||
to OpenVINO IR format. OpenVINO supports PyTorch via Model conversion
|
||
API. To convert the PyTorch model to OpenVINO IR format we will use
|
||
``ov.convert_model`` of `model conversion
|
||
API <https://docs.openvino.ai/2024/openvino-workflow/model-preparation.html>`__.
|
||
The ``ov.convert_model`` Python function returns an OpenVINO Model
|
||
object ready to load on the device and start making predictions.
|
||
|
||
Our model consist from 2 parts - image encoder and text encoder that can
|
||
be used separately. Let’s convert each part to OpenVINO.
|
||
|
||
.. code:: ipython3
|
||
|
||
import types
|
||
import torch.nn.functional as F
|
||
|
||
|
||
def se_block_forward(self, inputs):
|
||
"""Apply forward pass."""
|
||
b, c, h, w = inputs.size()
|
||
x = F.avg_pool2d(inputs, kernel_size=[8, 8])
|
||
x = self.reduce(x)
|
||
x = F.relu(x)
|
||
x = self.expand(x)
|
||
x = torch.sigmoid(x)
|
||
x = x.view(-1, c, 1, 1)
|
||
return inputs * x
|
||
|
||
.. code:: ipython3
|
||
|
||
import openvino as ov
|
||
import gc
|
||
|
||
ov_models_dir = Path("ov_models")
|
||
ov_models_dir.mkdir(exist_ok=True)
|
||
|
||
image_encoder_path = ov_models_dir / f"{model_checkpoint.value}_im_encoder.xml"
|
||
|
||
if not image_encoder_path.exists():
|
||
if "mobileclip_s" in model_name:
|
||
model.image_encoder.model.conv_exp.se.forward = types.MethodType(se_block_forward, model.image_encoder.model.conv_exp.se)
|
||
model.forward = model.encode_image
|
||
ov_image_encoder = ov.convert_model(model, example_input=image_tensor, input=[-1, 3, image_tensor.shape[2], image_tensor.shape[3]])
|
||
ov.save_model(ov_image_encoder, image_encoder_path)
|
||
del ov_image_encoder
|
||
gc.collect()
|
||
|
||
text_encoder_path = ov_models_dir / f"{model_checkpoint.value}_text_encoder.xml"
|
||
|
||
if not text_encoder_path.exists():
|
||
model.forward = model.encode_text
|
||
ov_text_encoder = ov.convert_model(model, example_input=text, input=[-1, text.shape[1]])
|
||
ov.save_model(ov_text_encoder, text_encoder_path)
|
||
del ov_text_encoder
|
||
gc.collect()
|
||
|
||
del model
|
||
gc.collect();
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-632/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/mobileclip/modules/common/transformer.py:125: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
|
||
if seq_len != self.num_embeddings:
|
||
|
||
|
||
Run OpenVINO model inference
|
||
----------------------------
|
||
|
||
|
||
|
||
Select device for image encoder
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
core = ov.Core()
|
||
|
||
import ipywidgets as widgets
|
||
|
||
device = widgets.Dropdown(
|
||
options=core.available_devices + ["AUTO"],
|
||
value='AUTO',
|
||
description='Device:',
|
||
disabled=False,
|
||
)
|
||
|
||
device
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_compiled_image_encoder = core.compile_model(image_encoder_path, device.value)
|
||
ov_compiled_image_encoder(image_tensor);
|
||
|
||
Select device for text encoder
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
device
|
||
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
ov_compiled_text_encoder = core.compile_model(text_encoder_path, device.value)
|
||
ov_compiled_text_encoder(text);
|
||
|
||
Perform search
|
||
~~~~~~~~~~~~~~
|
||
|
||
|
||
|
||
.. code:: ipython3
|
||
|
||
image_encoding_start = time.perf_counter()
|
||
image_features = torch.from_numpy(ov_compiled_image_encoder(image_tensor)[0])
|
||
image_encoding_end = time.perf_counter()
|
||
print(f"Image encoding took {image_encoding_end - image_encoding_start:.3} ms")
|
||
text_encoding_start = time.perf_counter()
|
||
text_features = torch.from_numpy(ov_compiled_text_encoder(text)[0])
|
||
text_encoding_end = time.perf_counter()
|
||
print(f"Text encoding took {text_encoding_end - text_encoding_start:.3} ms")
|
||
image_features /= image_features.norm(dim=-1, keepdim=True)
|
||
text_features /= text_features.norm(dim=-1, keepdim=True)
|
||
|
||
image_probs = (100.0 * text_features @ image_features.T).softmax(dim=-1)
|
||
selected_image = [torch.argmax(image_probs).item()]
|
||
|
||
visualize_result(images, input_labels[0], selected_image);
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Image encoding took 0.0306 ms
|
||
Text encoding took 0.0058 ms
|
||
|
||
|
||
|
||
.. image:: 289-mobileclip-video-search-with-output_files/289-mobileclip-video-search-with-output_25_1.png
|
||
|
||
|
||
Interactive Demo
|
||
----------------
|
||
|
||
|
||
|
||
In this part, you can try different supported by tutorial models in
|
||
searching frames in the video by text query or image. Upload v
|
||
|
||
.. code:: ipython3
|
||
|
||
import altair as alt
|
||
import cv2
|
||
import gradio as gr
|
||
import pandas as pd
|
||
import torch
|
||
from PIL import Image
|
||
from torch.utils.data import DataLoader, Dataset
|
||
from torchvision.transforms.functional import to_pil_image, to_tensor
|
||
from torchvision.transforms import (
|
||
CenterCrop,
|
||
Compose,
|
||
InterpolationMode,
|
||
Resize,
|
||
ToTensor,
|
||
)
|
||
from open_clip.transform import image_transform
|
||
|
||
|
||
current_device = device.value
|
||
current_model = image_encoder_path.name.split("_im_encoder")[0]
|
||
|
||
available_converted_models = [model_file.name.split("_im_encoder")[0] for model_file in ov_models_dir.glob("*_im_encoder.xml")]
|
||
available_devices = list(core.available_devices) + ["AUTO"]
|
||
|
||
download_file("https://github.com/intel-iot-devkit/sample-videos/raw/master/car-detection.mp4", directory=sample_path)
|
||
download_file("https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/video/Coco%20Walking%20in%20Berkeley.mp4", directory=sample_path, filename="coco.mp4")
|
||
|
||
def get_preprocess_and_tokenizer(model_name):
|
||
if "mobileclip" in model_name:
|
||
resolution = supported_models["MobileCLIP"][model_name]["image_size"]
|
||
resize_size = resolution
|
||
centercrop_size = resolution
|
||
aug_list = [
|
||
Resize(
|
||
resize_size,
|
||
interpolation=InterpolationMode.BILINEAR,
|
||
),
|
||
CenterCrop(centercrop_size),
|
||
ToTensor(),
|
||
]
|
||
preprocess = Compose(aug_list)
|
||
tokenizer = mobileclip.get_tokenizer(supported_models["MobileCLIP"][model_name]["model_name"])
|
||
else:
|
||
model_configs = supported_models["SigLIP"] if "siglip" in model_name else supported_models["CLIP"]
|
||
resize_size = model_configs[model_name]["image_size"]
|
||
preprocess = image_transform((resize_size, resize_size), is_train=False, resize_mode="longest")
|
||
tokenizer = open_clip.get_tokenizer(model_configs[model_name]["model_name"])
|
||
|
||
return preprocess, tokenizer
|
||
|
||
|
||
def run(
|
||
path: str,
|
||
text_search: str,
|
||
image_search: Image.Image,
|
||
model_name: str,
|
||
device: str,
|
||
thresh: float,
|
||
stride: int,
|
||
batch_size: int,
|
||
):
|
||
|
||
assert path, "An input video should be provided"
|
||
assert (
|
||
text_search is not None or image_search is not None
|
||
), "A text or image query should be provided"
|
||
global current_model
|
||
global current_device
|
||
global preprocess
|
||
global tokenizer
|
||
global ov_compiled_image_encoder
|
||
global ov_compiled_text_encoder
|
||
|
||
|
||
if current_model != model_name or device != current_device:
|
||
ov_compiled_image_encoder = core.compile_model(ov_models_dir / f"{model_name}_im_encoder.xml", device)
|
||
ov_compiled_text_encoder = core.compile_model(ov_models_dir / f"{model_name}_text_encoder.xml", device)
|
||
preprocess, tokenizer = get_preprocess_and_tokenizer(model_name)
|
||
current_model = model_name
|
||
current_device = device
|
||
# Load video
|
||
dataset = LoadVideo(path, transforms=preprocess, vid_stride=stride)
|
||
dataloader = DataLoader(
|
||
dataset, batch_size=batch_size, shuffle=False, num_workers=0
|
||
)
|
||
|
||
# Get image query features
|
||
if image_search:
|
||
image = preprocess(image_search).unsqueeze(0)
|
||
query_features = torch.from_numpy(ov_compiled_image_encoder(image)[0])
|
||
query_features /= query_features.norm(dim=-1, keepdim=True)
|
||
# Get text query features
|
||
else:
|
||
# Tokenize search phrase
|
||
text = tokenizer([text_search])
|
||
# Encode text query
|
||
query_features = torch.from_numpy(ov_compiled_text_encoder(text)[0])
|
||
query_features /= query_features.norm(dim=-1, keepdim=True)
|
||
# Encode each frame and compare with query features
|
||
matches = []
|
||
matches_probs = []
|
||
res = pd.DataFrame(columns=["Frame", "Timestamp", "Similarity"])
|
||
for image, orig, frame, timestamp in dataloader:
|
||
with torch.no_grad():
|
||
image_features = torch.from_numpy(ov_compiled_image_encoder(image)[0])
|
||
|
||
image_features /= image_features.norm(dim=-1, keepdim=True)
|
||
probs = query_features.cpu().numpy() @ image_features.cpu().numpy().T
|
||
probs = probs[0]
|
||
|
||
# Save frame similarity values
|
||
df = pd.DataFrame(
|
||
{
|
||
"Frame": frame.tolist(),
|
||
"Timestamp": torch.round(timestamp / 1000, decimals=2).tolist(),
|
||
"Similarity": probs.tolist(),
|
||
}
|
||
)
|
||
res = pd.concat([res, df])
|
||
|
||
# Check if frame is over threshold
|
||
for i, p in enumerate(probs):
|
||
if p > thresh:
|
||
matches.append(to_pil_image(orig[i]))
|
||
matches_probs.append(p)
|
||
|
||
print(f"Frames: {frame.tolist()} - Probs: {probs}")
|
||
|
||
# Create plot of similarity values
|
||
lines = (
|
||
alt.Chart(res)
|
||
.mark_line(color="firebrick")
|
||
.encode(
|
||
alt.X("Timestamp", title="Timestamp (seconds)"),
|
||
alt.Y("Similarity", scale=alt.Scale(zero=False)),
|
||
)
|
||
).properties(width=600)
|
||
rule = alt.Chart().mark_rule(strokeDash=[6, 3], size=2).encode(y=alt.datum(thresh))
|
||
|
||
selected_frames = np.argsort(-1 * np.array(matches_probs))[:20]
|
||
matched_sorted_frames = [matches[idx] for idx in selected_frames]
|
||
|
||
|
||
return lines + rule, matched_sorted_frames # Only return up to 20 images to not crash the UI
|
||
|
||
|
||
class LoadVideo(Dataset):
|
||
def __init__(self, path, transforms, vid_stride=1):
|
||
|
||
self.transforms = transforms
|
||
self.vid_stride = vid_stride
|
||
self.cur_frame = 0
|
||
self.cap = cv2.VideoCapture(path)
|
||
self.total_frames = int(
|
||
self.cap.get(cv2.CAP_PROP_FRAME_COUNT) / self.vid_stride
|
||
)
|
||
|
||
def __getitem__(self, _):
|
||
# Read video
|
||
# Skip over frames
|
||
for _ in range(self.vid_stride):
|
||
self.cap.grab()
|
||
self.cur_frame += 1
|
||
|
||
# Read frame
|
||
_, img = self.cap.retrieve()
|
||
timestamp = self.cap.get(cv2.CAP_PROP_POS_MSEC)
|
||
|
||
# Convert to PIL
|
||
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
|
||
img = Image.fromarray(np.uint8(img))
|
||
|
||
# Apply transforms
|
||
img_t = self.transforms(img)
|
||
|
||
return img_t, to_tensor(img), self.cur_frame, timestamp
|
||
|
||
def __len__(self):
|
||
return self.total_frames
|
||
|
||
|
||
|
||
desc_text = """
|
||
Search the content's of a video with a text description.
|
||
__Note__: Long videos (over a few minutes) may cause UI performance issues.
|
||
"""
|
||
text_app = gr.Interface(
|
||
description=desc_text,
|
||
fn=run,
|
||
inputs=[
|
||
gr.Video(label="Video"),
|
||
gr.Textbox(label="Text Search Query"),
|
||
gr.Image(label="Image Search Query", visible=False),
|
||
gr.Dropdown(label="Model", choices=available_converted_models, value=model_checkpoint.value),
|
||
gr.Dropdown(label="Device", choices=available_devices, value=device.value),
|
||
gr.Slider(label="Threshold", maximum=1.0, value=0.2),
|
||
gr.Slider(label="Frame-rate Stride", value=4, step=1),
|
||
gr.Slider(label="Batch Size", value=4, step=1),
|
||
],
|
||
outputs=[
|
||
gr.Plot(label="Similarity Plot"),
|
||
gr.Gallery(label="Matched Frames", columns=2, object_fit="contain", height="auto")
|
||
],
|
||
examples=[[sample_path / "car-detection.mp4", "white car"]],
|
||
allow_flagging="never",
|
||
)
|
||
|
||
desc_image = """
|
||
Search the content's of a video with an image query.
|
||
__Note__: Long videos (over a few minutes) may cause UI performance issues.
|
||
"""
|
||
image_app = gr.Interface(
|
||
description=desc_image,
|
||
fn=run,
|
||
inputs=[
|
||
gr.Video(label="Video"),
|
||
gr.Textbox(label="Text Search Query", visible=False),
|
||
gr.Image(label="Image Search Query", type="pil"),
|
||
gr.Dropdown(label="Model", choices=available_converted_models, value=model_checkpoint.value),
|
||
gr.Dropdown(label="Device", choices=available_devices, value=device.value),
|
||
gr.Slider(label="Threshold", maximum=1.0, value=0.2),
|
||
gr.Slider(label="Frame-rate Stride", value=4, step=1),
|
||
gr.Slider(label="Batch Size", value=4, step=1),
|
||
],
|
||
outputs=[
|
||
gr.Plot(label="Similarity Plot"),
|
||
gr.Gallery(label="Matched Frames", columns=2, object_fit="contain", height="auto")
|
||
],
|
||
allow_flagging="never",
|
||
examples=[[sample_path / "coco.mp4", None, sample_path / "dog.png"]]
|
||
)
|
||
demo = gr.TabbedInterface(
|
||
interface_list=[text_app, image_app],
|
||
tab_names=["Text Query Search", "Image Query Search"],
|
||
title="CLIP Video Content Search",
|
||
)
|
||
|
||
|
||
try:
|
||
demo.launch(debug=False)
|
||
except Exception:
|
||
demo.launch(share=True, debug=False)
|
||
# if you are launching remotely, specify server_name and server_port
|
||
# demo.launch(server_name='your server name', server_port='server port in int')
|
||
# Read more in the docs: https://gradio.app/docs/
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/car-detection.mp4: 0%| | 0.00/2.68M [00:00<?, ?B/s]
|
||
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
data/coco.mp4: 0%| | 0.00/877k [00:00<?, ?B/s]
|
||
|
||
|
||
.. parsed-literal::
|
||
|
||
Running on local URL: http://127.0.0.1:7860
|
||
|
||
To create a public link, set `share=True` in `launch()`.
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|