From 81d061fc70517e1a1d479109bdc69f224bcd030b Mon Sep 17 00:00:00 2001 From: Tatiana Savina Date: Thu, 29 Feb 2024 17:41:04 +0100 Subject: [PATCH] [DOCS] Links fixes (#23172) ### Details: - *item1* - *...* ### Tickets: - *ticket-id* --- .../documentation/openvino-ecosystem.rst | 22 +- .../openvino-security-add-on.rst | 442 +++++++++--------- .../openvino-training-extensions.rst | 14 +- .../openvino-workflow/deployment-locally.rst | 2 +- .../local-distribution-libraries.rst | 19 +- ...al-binary-size-conditional-compilation.rst | 8 +- .../generative-ai-models-guide.rst | 9 - .../compressing-models-during-training.rst | 8 +- .../filter-pruning.rst | 104 ++--- .../quantization-aware-training.rst | 108 ++--- .../openvino-workflow/model-preparation.rst | 3 +- .../conversion-parameters.rst | 2 +- .../convert-model-paddle.rst | 2 +- .../convert-model-pytorch.rst | 8 +- .../model-preparation/convert-model-to-ir.rst | 30 +- 15 files changed, 383 insertions(+), 398 deletions(-) diff --git a/docs/articles_en/documentation/openvino-ecosystem.rst b/docs/articles_en/documentation/openvino-ecosystem.rst index 49a40dcca0b..77b317ddaba 100644 --- a/docs/articles_en/documentation/openvino-ecosystem.rst +++ b/docs/articles_en/documentation/openvino-ecosystem.rst @@ -26,10 +26,10 @@ A suite of advanced algorithms for Neural Network inference optimization with mi More resources: -* :doc:`Documentation <../openvino-workflow/model-optimization-guide/compressing-models-during-training>` -* `GitHub `__ -* `PyPI `__ -* `Conda Forge `__ +* :doc:`Documentation <../openvino-workflow/model-optimization-guide/compressing-models-during-training>` +* `GitHub `__ +* `PyPI `__ +* `Conda Forge `__ **OpenVINO™ Training Extensions** @@ -40,27 +40,27 @@ More resources: * :doc:`Overview ` * `GitHub `__ -* `Documentation `__ +* `Documentation `__ **OpenVINO™ Security Add-on** -A solution for Model Developers and Independent Software Vendors to use secure packaging and secure model execution. +A solution for Model Developers and Independent Software Vendors to use secure packaging and secure model execution. More resources: * :doc:`Documentation ` -* `GitHub `__ +* `GitHub `__ **Dataset Management Framework (Datumaro)** A framework and CLI tool to build, transform, and analyze datasets. More resources: - + * :doc:`Overview ` -* `PyPI `__ -* `GitHub `__ -* `Documentation `__ +* `PyPI `__ +* `GitHub `__ +* `Documentation `__ diff --git a/docs/articles_en/documentation/openvino-ecosystem/openvino-security-add-on.rst b/docs/articles_en/documentation/openvino-ecosystem/openvino-security-add-on.rst index 961956cf227..ba87296fbe4 100644 --- a/docs/articles_en/documentation/openvino-ecosystem/openvino-security-add-on.rst +++ b/docs/articles_en/documentation/openvino-ecosystem/openvino-security-add-on.rst @@ -10,9 +10,9 @@ OpenVINO™ Security Add-on This guide provides instructions for people who use the OpenVINO™ Security Add-on to create, distribute, and use models that are created with the OpenVINO™ toolkit: -* **Model Developer**: The Model Developer interacts with the Independent Software Vendor to control the User access to models. This document shows you how to setup hardware and virtual machines to use the OpenVINO™ Security Add-on to define access control to your OpenVINO™ models and then provide the access controlled models to the users. -* **Independent Software Vendor**: Use this guide for instructions to use the OpenVINO™ Security Add-on to validate license for access controlled models that are provided to your customers (users). -* **User**: This document includes instructions for end users who need to access and run access controlled models through the OpenVINO™ Security Add-on. +* **Model Developer**: The Model Developer interacts with the Independent Software Vendor to control the User access to models. This document shows you how to setup hardware and virtual machines to use the OpenVINO™ Security Add-on to define access control to your OpenVINO™ models and then provide the access controlled models to the users. +* **Independent Software Vendor**: Use this guide for instructions to use the OpenVINO™ Security Add-on to validate license for access controlled models that are provided to your customers (users). +* **User**: This document includes instructions for end users who need to access and run access controlled models through the OpenVINO™ Security Add-on. In this release, one person performs the role of both the Model Developer and the Independent Software Vendor. Therefore, this document provides instructions to configure one system for these two roles and one system for the User role. This document also provides a way for the same person to play the role of the Model Developer, Independent Software Vendor, and User to let you see how the OpenVINO™ Security Add-on functions from the User perspective. @@ -21,27 +21,27 @@ Overview The OpenVINO™ Security Add-on works with the :doc:`OpenVINO™ Model Server <../../ovms_what_is_openvino_model_server>` on Intel® architecture. Together, the OpenVINO™ Security Add-on and the OpenVINO™ Model Server provide a way for Model Developers and Independent Software Vendors to use secure packaging and secure model execution to enable access control to the OpenVINO™ models, and for model Users to run inference within assigned limits. -The OpenVINO™ Security Add-on consists of three components that run in Kernel-based Virtual Machines (KVMs). These components provide a way to run security-sensitive operations in an isolated environment. A brief description of the three components are as follows. Click each triangled line for more information about each. +The OpenVINO™ Security Add-on consists of three components that run in Kernel-based Virtual Machines (KVMs). These components provide a way to run security-sensitive operations in an isolated environment. A brief description of the three components are as follows. Click each triangled line for more information about each. .. dropdown:: OpenVINO™ Security Add-on Tool: As a Model Developer or Independent Software Vendor, you use the OpenVINO™ Security Add-on Tool (``ovsatool``) to generate a access controlled model and master license. - - - The Model Developer generates a access controlled model from the OpenVINO™ toolkit output. The access controlled model uses the model's Intermediate Representation (IR) files to create a access controlled output file archive that are distributed to Model Users. The Developer can also put the archive file in long-term storage or back it up without additional security. - - The Model Developer uses the OpenVINO™ Security Add-on Tool (ovsatool) to generate and manage cryptographic keys and related collateral for the access controlled models. Cryptographic material is only available in a virtual machine (VM) environment. The OpenVINO™ Security Add-on key management system lets the Model Developer to get external Certificate Authorities to generate certificates to add to a key-store. - - The Model Developer generates user-specific licenses in a JSON format file for the access controlled model. The Model Developer can define global or user-specific licenses and attach licensing policies to the licenses. For example, the Model Developer can add a time limit for a model or limit the number of times a user can run a model. + + - The Model Developer generates a access controlled model from the OpenVINO™ toolkit output. The access controlled model uses the model's Intermediate Representation (IR) files to create a access controlled output file archive that are distributed to Model Users. The Developer can also put the archive file in long-term storage or back it up without additional security. + - The Model Developer uses the OpenVINO™ Security Add-on Tool (ovsatool) to generate and manage cryptographic keys and related collateral for the access controlled models. Cryptographic material is only available in a virtual machine (VM) environment. The OpenVINO™ Security Add-on key management system lets the Model Developer to get external Certificate Authorities to generate certificates to add to a key-store. + - The Model Developer generates user-specific licenses in a JSON format file for the access controlled model. The Model Developer can define global or user-specific licenses and attach licensing policies to the licenses. For example, the Model Developer can add a time limit for a model or limit the number of times a user can run a model. .. dropdown:: OpenVINO™ Security Add-on License Service: Use the OpenVINO™ Security Add-on License Service to verify user parameters. - The Independent Software Vendor hosts the OpenVINO™ Security Add-on License Service, which responds to license validation requests when a user attempts to load a access controlled model in a model server. The licenses are registered with the OpenVINO™ Security Add-on License Service. - - When a user loads the model, the OpenVINO™ Security Add-on Runtime contacts the License Service to make sure the license is valid and within the parameters that the Model Developer defined with the OpenVINO™ Security Add-on Tool (``ovsatool``). The user must be able to reach the Independent Software Vendor's License Service over the Internet. + - When a user loads the model, the OpenVINO™ Security Add-on Runtime contacts the License Service to make sure the license is valid and within the parameters that the Model Developer defined with the OpenVINO™ Security Add-on Tool (``ovsatool``). The user must be able to reach the Independent Software Vendor's License Service over the Internet. .. dropdown:: OpenVINO™ Security Add-on Runtime: Users install and use the OpenVINO™ Security Add-on Runtime on a virtual machine. - Users host the OpenVINO™ Security Add-on Runtime component in a virtual machine. + Users host the OpenVINO™ Security Add-on Runtime component in a virtual machine. - Externally from the OpenVINO™ Security Add-on, the User adds the access controlled model to the OpenVINO™ Model Server config file. The OpenVINO™ Model Server attempts to load the model in memory. At this time, the OpenVINO™ Security Add-on Runtime component validates the user's license for the access controlled model against information stored in the License Service provided by the Independent Software Vendor. + Externally from the OpenVINO™ Security Add-on, the User adds the access controlled model to the OpenVINO™ Model Server config file. The OpenVINO™ Model Server attempts to load the model in memory. At this time, the OpenVINO™ Security Add-on Runtime component validates the user's license for the access controlled model against information stored in the License Service provided by the Independent Software Vendor. - After the license is successfully validated, the OpenVINO™ Model Server loads the model and services the inference requests. + After the license is successfully validated, the OpenVINO™ Model Server loads the model and services the inference requests. @@ -54,7 +54,7 @@ The binding between SWTPM (vTPM used in guest VM) and HW TPM (TPM on the host) i About the Installation ###################### -The Model Developer, Independent Software Vendor, and User each must prepare one physical hardware machine and one Kernel-based Virtual Machine (KVM). In addition, each person must prepare a Guest Virtual Machine (Guest VM) for each role that person plays. +The Model Developer, Independent Software Vendor, and User each must prepare one physical hardware machine and one Kernel-based Virtual Machine (KVM). In addition, each person must prepare a Guest Virtual Machine (Guest VM) for each role that person plays. For example: @@ -105,68 +105,68 @@ Step 1: Set up Packages on the Host Machine Begin this step on the Intel® Core™ or Xeon® processor machine that meets the :ref:`prerequisites `. -.. note:: - +.. note:: + As an alternative to manually following steps 1 - 11, you can run the script ``install_host_deps.sh`` in the ``Scripts/reference directory`` under the OpenVINO™ Security Add-on repository. The script stops with an error message if it identifies any issues. If the script halts due to an error, correct the issue that caused the error and restart the script. The script runs for several minutes and provides progress information. 1. Test for Trusted Platform Module (TPM) support: .. code-block:: sh - - dmesg | grep -i TPM - + + dmesg | grep -i TPM + The output indicates TPM availability in the kernel boot logs. Look for presence of the following devices to indicate TPM support is available: - + * ``/dev/tpm0`` * ``/dev/tpmrm0`` If you do not see this information, your system does not meet the :ref:`prerequisites ` to use the OpenVINO™ Security Add-on. 2. Make sure hardware virtualization support is enabled in the BIOS: - + .. code-block:: sh - - kvm-ok - + + kvm-ok + The output should show: - ``INFO: /dev/kvm exists`` + ``INFO: /dev/kvm exists`` ``KVM acceleration can be used`` - + If your output is different, modify your BIOS settings to enable hardware virtualization. - + If the ``kvm-ok`` command is not present, install it: .. code-block:: sh - + sudo apt install -y cpu-checker - -3. Install the Kernel-based Virtual Machine (KVM) and QEMU packages. + +3. Install the Kernel-based Virtual Machine (KVM) and QEMU packages. .. code-block:: sh - - sudo apt install qemu qemu-kvm libvirt-bin bridge-utils virt-manager - + + sudo apt install qemu qemu-kvm libvirt-bin bridge-utils virt-manager + 4. Check the QEMU version: .. code-block:: sh - - qemu-system-x86_64 --version - + + qemu-system-x86_64 --version + If the response indicates a QEMU version lower than 2.12.0 download, compile and install the latest QEMU version from `QEMU website `__. -5. Build and install the `libtpm package `__. +5. Build and install the `libtpm package `__. 6. Build and install the `swtpm package `__. 7. Add the ``swtpm`` package to the ``$PATH`` environment variable. 8. Install the software tool `tpm2-tss `__ . For installation information follow `here `__. 9. Install the software tool `tpm2-abmrd `__ . For installation information follow `here `__. 10. Install the `tpm2-tools `__ . For installation information follow `here `__. 11. Install the `Docker packages `__ . - - .. note:: - + + .. note:: + Regardless of whether you used the ``install_host_deps.sh`` script, complete step 12 to finish setting up the packages on the Host Machine. -12. If you are running behind a proxy, `set up a proxy for Docker `__. +12. If you are running behind a proxy, `set up a proxy for Docker `__. The following are installed and ready to use: * Kernel-based Virtual Machine (KVM) @@ -174,8 +174,8 @@ The following are installed and ready to use: * SW-TPM * HW-TPM support * Docker - -You're ready to configure the Host Machine for networking. + +You're ready to configure the Host Machine for networking. .. _setup_networking: @@ -192,26 +192,26 @@ In this step you prepare two network bridges: This example in this step uses the following names. Your configuration might use different names: * ``50-cloud-init.yaml`` as an example configuration file name. -* ``eno1`` as an example network interface name. +* ``eno1`` as an example network interface name. * ``br0`` as an example bridge name. * ``virbr0`` as an example bridge name. 1. Open the network configuration file for editing. This file is in ``/etc/netplan`` with a name like ``50-cloud-init.yaml`` 2. Look for these lines in the file: - + .. code-block:: sh - + network: ethernets: eno1: dhcp4: true dhcp-identifier: mac version: 2 - + 3. Change the existing lines and add the ``br0`` network bridge. These changes enable external network access: - + .. code-block:: sh - + network: ethernets: eno1: @@ -222,35 +222,35 @@ This example in this step uses the following names. Your configuration might use dhcp4: yes dhcp-identifier: mac version: 2 - + 4. Save and close the network configuration file. 5. Run two commands to activate the updated network configuration file. If you use ssh, you might lose network connectivity when issuing these commands. If so, reconnect to the network. .. code-block:: sh - + sudo netplan generate .. code-block:: sh - + sudo netplan apply A bridge is created and an IP address is assigned to the new bridge. 6. Verify the new bridge: - + .. code-block:: sh - + ip a | grep br0 - + The output looks similar to this and shows valid IP addresses: .. code-block:: sh - + 4: br0: mtu 1500 qdisc noqueue state UP group default qlen 1000inet 123.123.123.123/ brd 321.321.321.321 scope global dynamic br0 - + 7. Create a script named ``br0-qemu-ifup`` to bring up the ``br0`` interface. Add the following script contents: .. code-block:: sh - + #!/bin/sh nic=$1 if [ -f /etc/default/qemu-kvm ]; then @@ -259,11 +259,11 @@ This example in this step uses the following names. Your configuration might use switch=br0 ifconfig $nic 0.0.0.0 up brctl addif ${switch} $nic - + 8. Create a script named ``br0-qemu-ifdown`` to bring down the ``br0`` interface. Add the following script contents: .. code-block:: sh - + #!/bin/sh nic=$1 if [ -f /etc/default/qemu-kvm ]; then @@ -272,11 +272,11 @@ This example in this step uses the following names. Your configuration might use switch=br0 brctl delif $switch $nic ifconfig $nic 0.0.0.0 down - + 9. Create a script named ``virbr0-qemu-ifup`` to bring up the ``virbr0`` interface. Add the following script contents: - + .. code-block:: sh - + #!/bin/sh nic=$1 if [ -f /etc/default/qemu-kvm ]; then @@ -285,11 +285,11 @@ This example in this step uses the following names. Your configuration might use switch=virbr0 ifconfig $nic 0.0.0.0 up brctl addif ${switch} $nic - + 10. Create a script named ``virbr0-qemu-ifdown`` to bring down the ``virbr0`` interface. Add the following script contents: - + .. code-block:: sh - + #!/bin/sh nic=$1 if [ -f /etc/default/qemu-kvm ]; then @@ -298,9 +298,9 @@ This example in this step uses the following names. Your configuration might use switch=virbr0 brctl delif $switch $nic ifconfig $nic 0.0.0.0 down - -See the QEMU ../../documentation for more information about the QEMU network configuration. + +See the QEMU documentation for more information about the QEMU network configuration. Networking is set up on the Host Machine. Continue to the Step 3 to prepare a Guest VM for the combined role of Model Developer and Independent Software Vendor. @@ -316,22 +316,22 @@ Step 4: Set Up one Guest VM for the combined roles of Model Developer and Indepe For each separate role you play, you must prepare a virtual machine, called a Guest VM. Because in this release, the Model Developer and Independent Software Vendor roles are combined, these instructions guide you to set up one Guest VM, named ``ovsa_isv``. -Begin these steps on the Host Machine. +Begin these steps on the Host Machine. -As an option, you can use ``virsh`` and the virtual machine manager to create and bring up a Guest VM. See the ``libvirtd`` ../../documentation for instructions if you'd like to do this. +As an option, you can use ``virsh`` and the virtual machine manager to create and bring up a Guest VM. See the ``libvirtd`` documentation for instructions if you'd like to do this. 1. Download the **Server install ISO image for 64-bit PC (AMD64) computers** of `Ubuntu 18.04 `__ 2. Create an empty virtual disk image to serve as the Guest VM for your role as Model Developer and Independent Software Vendor: .. code-block:: sh - + sudo qemu-img create -f qcow2 /ovsa_isv_dev_vm_disk.qcow2 20G - + 3. Install Ubuntu 18.04 on the Guest VM. Name the Guest VM ``ovsa_isv``: - + .. code-block:: sh - + sudo qemu-system-x86_64 -m 8192 -enable-kvm \ -cpu host \ -drive if=virtio,file=/ovsa_isv_dev_vm_disk.qcow2,cache=none \ @@ -339,21 +339,21 @@ As an option, you can use ``virsh`` and the virtual machine manager to create an -device e1000,netdev=hostnet1,mac=52:54:00:d1:66:5f \ -netdev tap,id=hostnet1,script=/virbr0-qemu-ifup,downscript=/virbr0-qemu-ifdown \ -vnc :1 - + 4. Connect a VNC client with ``:1`` 5. Follow the prompts on the screen to finish installing the Guest VM. Name the VM as ``ovsa_isv_dev`` -6. Shut down the Guest VM. +6. Shut down the Guest VM. 7. Restart the Guest VM after removing the option of cdrom image: - + .. code-block:: sh - + sudo qemu-system-x86_64 -m 8192 -enable-kvm \ -cpu host \ -drive if=virtio,file=/ovsa_isv_dev_vm_disk.qcow2,cache=none \ -device e1000,netdev=hostnet1,mac=52:54:00:d1:66:5f \ -netdev tap,id=hostnet1,script=/virbr0-qemu-ifup,downscript=/virbr0-qemu-ifdown \ -vnc :1 - + 8. Choose ONE of these options to install additional required software: * **Option 1**: Use a script to install additional software @@ -362,55 +362,55 @@ As an option, you can use ``virsh`` and the virtual machine manager to create an 3. Shut down the Guest VM. * **Option 2** : Manually install additional software - 1. Install the software tool `tpm2-tss `__. + 1. Install the software tool `tpm2-tss `__. For installation information follow `here `__ - 2. Install the software tool `tpm2-abmrd `__. + 2. Install the software tool `tpm2-abmrd `__. For installation information follow `here `__ - 3. Install the `tpm2-tools `__. + 3. Install the `tpm2-tools `__. For installation information follow `here `__ 4. Install the `Docker packages `__ 5. Shut down the Guest VM. 9. On the host, create a directory to support the virtual TPM device and provision its certificates. Only ``root`` should have read/write permission to this directory: - + .. code-block:: sh - + sudo mkdir -p /var/OVSA/ sudo mkdir /var/OVSA/vtpm sudo mkdir /var/OVSA/vtpm/vtpm_isv_dev - + export XDG_CONFIG_HOME=~/.config /usr/share/swtpm/swtpm-create-user-config-files swtpm_setup --tpmstate /var/OVSA/vtpm/vtpm_isv_dev --create-ek-cert --create-platform-cert --overwrite --tpm2 --pcr-banks - - - .. note:: - + + .. note:: + For steps 10 and 11, you can copy and edit the script named ``start_ovsa_isv_dev_vm.sh`` in the ``Scripts/reference`` directory in the OpenVINO™ Security Add-on repository instead of manually running the commands. If using the script, select the script with ``isv`` in the file name regardless of whether you are playing the role of the Model Developer or the role of the Independent Software Vendor. Edit the script to point to the correct directory locations and increment ``vnc`` for each Guest VM. 10. Start the vTPM on Host, write the HW TPM data into its NVRAM and restart the vTPM for QEMU: - + .. code-block:: sh - + sudo swtpm socket --tpm2 --server port=8280 \ --ctrl type=tcp,port=8281 \ --flags not-need-init --tpmstate dir=/var/OVSA/vtpm/vtpm_isv_dev & - + sudo tpm2_startup --clear -T swtpm:port=8280 sudo tpm2_startup -T swtpm:port=8280 python3 /Scripts/host/OVSA_write_hwquote_swtpm_nvram.py 8280 sudo pkill -f vtpm_isv_dev - + swtpm socket --tpmstate dir=/var/OVSA/vtpm/vtpm_isv_dev \ --tpm2 \ --ctrl type=unixio,path=/var/OVSA/vtpm/vtpm_isv_dev/swtpm-sock \ --log level=20 - - + + 11. Start the Guest VM: - + .. code-block:: sh - + sudo qemu-system-x86_64 \ -cpu host \ -enable-kvm \ @@ -425,9 +425,9 @@ As an option, you can use ``virsh`` and the virtual machine manager to create an -tpmdev emulator,id=tpm0,chardev=chrtpm \ -device tpm-tis,tpmdev=tpm0 \ -vnc :1 - + Use the QEMU runtime options in the command to change the memory amount or CPU assigned to this Guest VM. - + 12. Use a VNC client to log on to the Guest VM at ``:1`` Step 5: Set Up one Guest VM for the User role @@ -438,40 +438,40 @@ Step 5: Set Up one Guest VM for the User role **Option 1: Copy and Rename the ovsa_isv_dev_vm_disk.qcow2 disk image** 1. Copy the ``ovsa_isv_dev_vm_disk.qcow2`` disk image to a new image named ``ovsa_runtime_vm_disk.qcow2``. You created the ``ovsa_isv_dev_vm_disk.qcow2`` disk image in :ref:`Step 4 `. - 2. Boot the new image. - 3. Change the hostname from ``ovsa_isv_dev`` to ``ovsa_runtime``. + 2. Boot the new image. + 3. Change the hostname from ``ovsa_isv_dev`` to ``ovsa_runtime``. .. code-block:: sh - + sudo hostnamectl set-hostname ovsa_runtime - + 4. Replace all instances of ``ovsa_isv_dev`` to ``ovsa_runtime`` in the new image. - + .. code-block:: sh - + sudo nano /etc/hosts - + 5. Change the ``/etc/machine-id``: .. code-block:: sh - + sudo rm /etc/machine-id systemd-machine-id-setup - + 6. Shut down the Guest VM. **Option 2: Manually create the Guest VM** 1. Create an empty virtual disk image: - + .. code-block:: sh - + sudo qemu-img create -f qcow2 /ovsa_ovsa_runtime_vm_disk.qcow2 20G - + 2. Install Ubuntu 18.04 on the Guest VM. Name the Guest VM ``ovsa_runtime``: .. code-block:: sh - + sudo qemu-system-x86_64 -m 8192 -enable-kvm \ -cpu host \ -drive if=virtio,file=/ovsa_ovsa_runtime_vm_disk.qcow2,cache=none \ @@ -479,66 +479,66 @@ Step 5: Set Up one Guest VM for the User role -device e1000,netdev=hostnet1,mac=52:54:00:d1:66:5f \ -netdev tap,id=hostnet1,script=/virbr0-qemu-ifup, downscript=/virbr0-qemu-ifdown \ -vnc :2 - + 3. Connect a VNC client with ``:2``. 4. Follow the prompts on the screen to finish installing the Guest VM. Name the Guest VM ``ovsa_runtime``. - 5. Shut down the Guest VM. + 5. Shut down the Guest VM. 6. Restart the Guest VM: .. code-block:: sh - + sudo qemu-system-x86_64 -m 8192 -enable-kvm \ -cpu host \ -drive if=virtio,file=/ovsa_ovsa_runtime_vm_disk.qcow2,cache=none \ -device e1000,netdev=hostnet1,mac=52:54:00:d1:66:5f \ -netdev tap,id=hostnet1,script=/virbr0-qemu-ifup, downscript=/virbr0-qemu-ifdown \ -vnc :2 - + 7. Choose **ONE** of these options to install additional required software: - + **Option 1: Use a script to install additional software** 1. Copy the script ``install_guest_deps.sh`` from the ``Scripts/reference`` directory of the OVSA repository to the Guest VM 2. Run the script. 3. Shut down the Guest VM. - + **Option 2: Manually install additional software** 1. Install the software tool `tpm2-tss `__ For installation information follow `here `__ 2. Install the software tool `tpm2-abmrd `__ For installation information follow `here `__ 3. Install the `tpm2-tools `__ For installation information follow `here `__ - 4. Install the `Docker packages `__ + 4. Install the `Docker packages `__ 5. Shut down the Guest VM. 2. Create a directory to support the virtual TPM device and provision its certificates. Only ``root`` should have read/write permission to this directory: .. code-block:: sh - + sudo mkdir /var/OVSA/vtpm/vtpm_runtime - + export XDG_CONFIG_HOME=~/.config /usr/share/swtpm/swtpm-create-user-config-files swtpm_setup --tpmstate /var/OVSA/vtpm/vtpm_runtime --create-ek-cert --create-platform-cert --overwrite --tpm2 --pcr-banks - - - .. note:: - + + .. note:: + For steps 3 and 4, you can copy and edit the script named ``start_ovsa_runtime_vm.sh`` in the ``Scripts/reference`` directory in the OpenVINO™ Security Add-on repository instead of manually running the commands. Edit the script to point to the correct directory locations and increment ``vnc`` for each Guest VM. This means that if you are creating a third Guest VM on the same Host Machine, change ``-vnc :2`` to ``-vnc :3`` 3. Start the vTPM, write the HW TPM data into its NVRAM and restart the vTPM for QEMU: .. code-block:: sh - + sudo swtpm socket --tpm2 --server port=8380 \ --ctrl type=tcp,port=8381 \ --flags not-need-init --tpmstate dir=/var/OVSA/vtpm/vtpm_runtime & - + sudo tpm2_startup --clear -T swtpm:port=8380 sudo tpm2_startup -T swtpm:port=8380 python3 /Scripts/host/OVSA_write_hwquote_swtpm_nvram.py 8380 sudo pkill -f vtpm_runtime - + swtpm socket --tpmstate dir=/var/OVSA/vtpm/vtpm_runtime \ --tpm2 \ --ctrl type=unixio,path=/var/OVSA/vtpm/vtpm_runtime/swtpm-sock \ @@ -547,7 +547,7 @@ Step 5: Set Up one Guest VM for the User role 4. Start the Guest VM in a new terminal: .. code-block:: sh - + sudo qemu-system-x86_64 \ -cpu host \ -enable-kvm \ @@ -562,7 +562,7 @@ Step 5: Set Up one Guest VM for the User role -tpmdev emulator,id=tpm0,chardev=chrtpm \ -device tpm-tis,tpmdev=tpm0 \ -vnc :2 - + Use the QEMU runtime options in the command to change the memory amount or CPU assigned to this Guest VM. 5. Use a VNC client to log on to the Guest VM at ``:`` where ```` corresponds to the vnc number in the ``start_ovsa_isv_vm.sh`` or in step 8. @@ -574,7 +574,7 @@ How to Build and Install the OpenVINO™ Security Add-on Software Follow the below steps to build and Install OpenVINO™ Security Add-on on host and different VMs. -Step 1: Build the OpenVINO™ Model Server image +Step 1: Build the OpenVINO™ Model Server image ++++++++++++++++++++++++++++++++++++++++++++++ Building OpenVINO™ Security Add-on depends on OpenVINO™ Model Server docker containers. Download and build OpenVINO™ Model Server first on the host. @@ -584,11 +584,11 @@ Building OpenVINO™ Security Add-on depends on OpenVINO™ Model Server docker 2. Build the `OpenVINO™ Model Server Docker images `__ .. code-block:: sh - + git clone https://github.com/openvinotoolkit/model_server.git cd model_server make docker_build - + Step 2: Build the software required for all roles +++++++++++++++++++++++++++++++++++++++++++++++++ @@ -598,13 +598,13 @@ This step is for the combined role of Model Developer and Independent Software V 1. Go to the top-level OpenVINO™ Security Add-on source directory cloned earlier. .. code-block:: sh - + cd security_addon - + 2. Build the OpenVINO™ Security Add-on: .. code-block:: sh - + make clean all sudo -s make package ``` @@ -616,29 +616,29 @@ This step is for the combined role of Model Developer and Independent Software V Step 3: Install the host software +++++++++++++++++++++++++++++++++ -This step is for the combined role of Model Developer and Independent Software Vendor, and the User. +This step is for the combined role of Model Developer and Independent Software Vendor, and the User. 1. Go to the ``release_files`` directory: - - .. code-block:: sh - - cd release_files - -2. Set up the path: - - .. code-block:: sh - - export OVSA_RELEASE_PATH=$PWD - -3. Install the OpenVINO™ Security Add-on Software on the Host Machine: .. code-block:: sh - + + cd release_files + +2. Set up the path: + + .. code-block:: sh + + export OVSA_RELEASE_PATH=$PWD + +3. Install the OpenVINO™ Security Add-on Software on the Host Machine: + + .. code-block:: sh + cd $OVSA_RELEASE_PATH tar xvfz ovsa-kvm-host.tar.gz cd ovsa-kvm-host ./install.sh - + If you are using more than one Host Machine repeat Step 3 on each. @@ -646,49 +646,49 @@ Step 4: Install the OpenVINO™ Security Add-on Model Developer / ISV Components ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ This step is for the combined role of Model Developer and Independent Software Vendor. References to the Guest VM are to ``ovsa_isv_dev``. - + 1. Log on to the Guest VM as ````. 2. Create the OpenVINO™ Security Add-on directory in the home directory .. code-block:: sh - + mkdir -p ~/OVSA - + 3. Go to the Host Machine, outside of the Guest VM. 4. Copy ``ovsa-developer.tar.gz`` from ``release_files`` to the Guest VM: .. code-block:: sh - + cd $OVSA_RELEASE_PATH scp ovsa-developer.tar.gz username@://OVSA - + 5. Go to the Guest VM. 6. Create ``ovsa`` user .. code-block:: sh - + sudo useradd -m ovsa sudo passwd ovsa - + 7. Install the software to the Guest VM: .. code-block:: sh - + cd ~/OVSA tar xvfz ovsa-developer.tar.gz cd ovsa-developer sudo ./install.sh - + 8. Start the license server on a separate terminal as ``ovsa`` user. - + .. code-block:: sh - + source /opt/ovsa/scripts/setupvars.sh cd /opt/ovsa/bin ./license_server - - .. note:: - + + .. note:: + If you are behind a firewall, check and set your proxy settings to ensure the license server is able to validate the certificates. Step 5: Install the OpenVINO™ Security Add-on Model Hosting Component @@ -696,51 +696,51 @@ Step 5: Install the OpenVINO™ Security Add-on Model Hosting Component This step is for the User. References to the Guest VM are to ``ovsa_runtime``. -The Model Hosting components install the OpenVINO™ Security Add-on Runtime Docker container based on OpenVINO™ Model Server NGINX Docker to host a access controlled model. - +The Model Hosting components install the OpenVINO™ Security Add-on Runtime Docker container based on OpenVINO™ Model Server NGINX Docker to host a access controlled model. + 1. Log on to the Guest VM as ````. 2. Create the OpenVINO™ Security Add-on directory in the home directory .. code-block:: sh - + mkdir -p ~/OVSA - + 3. While on the Host Machine copy the ovsa-model-hosting.tar.gz from release_files to the Guest VM: .. code-block:: sh - + cd $OVSA_RELEASE_PATH scp ovsa-model-hosting.tar.gz username@://OVSA - + 4. Go to the Guest VM. 5. Create ``ovsa`` user .. code-block:: sh - + sudo useradd -m ovsa sudo passwd ovsa sudo usermod -aG docker ovsa - + 6. Install the software to the Guest VM: .. code-block:: sh - + cd ~/OVSA tar xvfz ovsa-model-hosting.tar.gz cd ovsa-model-hosting sudo ./install.sh - + How to Use the OpenVINO™ Security Add-on ######################################## This section requires interactions between the Model Developer/Independent Software vendor and the User. All roles must complete all applicable :ref:`set up steps ` and :ref:`installation steps ` before beginning this section. -This document uses the :ref:`face-detection-retail-0004 <../../omz_models_model_face_detection_retail_0044>` model as an example. +This document uses the :ref:`face-detection-retail-0004 <../../omz_models_model_face_detection_retail_0044>` model as an example. The following figure describes the interactions between the Model Developer, Independent Software Vendor, and User. -.. tip:: +.. tip:: The Model Developer/Independent Software Vendor and User roles are related to virtual machine use and one person might fill the tasks required by multiple roles. In this document the tasks of Model Developer and Independent Software Vendor are combined and use the Guest VM named ``ovsa_isv``. It is possible to have all roles set up on the same Host Machine. @@ -759,7 +759,7 @@ Step 1: Set up the artefacts directory Create a directory named artefacts. This directory will hold artefacts required to create licenses: .. code-block:: sh - + mkdir -p ~/OVSA/artefacts cd ~/OVSA/artefacts export OVSA_DEV_ARTEFACTS=$PWD @@ -773,23 +773,23 @@ Step 2: Create a key store and add a certificate to it This example uses a self-signed certificate for demonstration purposes. In a production environment, use CSR files to request for a CA-signed certificate. .. code-block:: sh - + cd $OVSA_DEV_ARTEFACTS /opt/ovsa/bin/ovsatool keygen -storekey -t ECDSA -n Intel -k isv_keystore -r isv_keystore.csr -e "/C=IN/CN=localhost" - + Below two files are created along with the keystore file: - - ``isv_keystore.csr``- A Certificate Signing Request (CSR) + - ``isv_keystore.csr``- A Certificate Signing Request (CSR) - ``isv_keystore.csr.crt`` - A self-signed certificate - + In a production environment, send ``isv_keystore.csr`` to a CA to request a CA-signed certificate. - + 3. Add the certificate to the key store .. code-block:: sh - + /opt/ovsa/bin/ovsatool keygen -storecert -c isv_keystore.csr.crt -k isv_keystore - - + + Step 3: Create the model ------------------------ @@ -798,7 +798,7 @@ This example uses ``curl`` to download the ``face-detection-retail-004`` model f Download a model from the Model Zoo: .. code-block:: sh - + curl --create-dirs https://download.01.org/opencv/2021/openvinotoolkit/2021.1/open_../legacy-features/model-zoo/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.xml https://download.01.org/opencv/2021/openvinotoolkit/2021.1/open_../legacy-features/model-zoo/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.bin -o model/face-detection-retail-0004.xml -o model/face-detection-retail-0004.bin The model is downloaded to the ``OVSA_DEV_ARTEFACTS/model`` directory @@ -809,7 +809,7 @@ Step 4: Define access control for the model and create a master license for it Define and enable the model access control and master license: .. code-block:: sh - + uuid=$(uuidgen) /opt/ovsa/bin/ovsatool controlAccess -i model/face-detection-retail-0004.xml model/face-detection-retail-0004.bin -n "face detection" -d "face detection retail" -v 0004 -p face_detection_model.dat -m face_detection_model.masterlic -k isv_keystore -g $uuid @@ -823,10 +823,10 @@ Use the runtime reference TCB to create a customer license for the access contro Generate the reference TCB for the runtime .. code-block:: sh - + /opt/ovsa/bin/ovsaruntime gen-tcb-signature -n "Face Detect @ Runtime VM" -v "1.0" -f face_detect_runtime_vm.tcb -k isv_keystore - + Step 6: Publish the access controlled Model and Runtime Reference TCB --------------------------------------------------------------------- @@ -836,18 +836,18 @@ Step 7: Receive a User Request ------------------------------ 1. Obtain artefacts from the User who needs access to a access controlled model: - + * Customer certificate from the customer's key store. * Other information that apply to your licensing practices, such as the length of time the user needs access to the model 2. Create a customer license configuration .. code-block:: sh - + cd $OVSA_DEV_ARTEFACTS /opt/ovsa/bin/ovsatool licgen -t TimeLimit -l30 -n "Time Limit License Config" -v 1.0 -u ":" /opt/ovsa/certs/server.crt -k isv_keystore -o 30daylicense.config - - .. note:: + + .. note:: The parameter /opt/ovsa/certs/server.crt contains the certificate used by the License Server. The server certificate will be added to the customer license and validated during use. Refer to `OpenVINO™ Security Add-on License Server Certificate Pinning `__ @@ -857,15 +857,15 @@ Step 7: Receive a User Request cd $OVSA_DEV_ARTEFACTS /opt/ovsa/bin/ovsatool sale -m face_detection_model.masterlic -k isv_keystore -l 30daylicense.config -t face_detect_runtime_vm.tcb -p custkeystore.csr.crt -c face_detection_model.lic - - + + 4. Update the license server database with the license. .. code-block:: sh cd /opt/ovsa/DB python3 ovsa_store_customer_lic_cert_db.py ovsa.db $OVSA_DEV_ARTEFACTS/face_detection_model.lic $OVSA_DEV_ARTEFACTS/custkeystore.csr.crt - + 5. Provide these files to the User: @@ -883,12 +883,12 @@ Step 1: Setup up the artefacts directory 1. Create a directory named artefacts. This directory will hold artefacts required to create licenses: .. code-block:: sh - + mkdir -p ~/OVSA/artefacts cd ~/OVSA/artefacts export OVSA_RUNTIME_ARTEFACTS=$PWD source /opt/ovsa/scripts/setupvars.sh - + Step 2: Add a CA-Signed Certificate to a Key Store -------------------------------------------------- @@ -896,10 +896,10 @@ Step 2: Add a CA-Signed Certificate to a Key Store 1. Generate a Customer key store file: .. code-block:: sh - + cd $OVSA_RUNTIME_ARTEFACTS /opt/ovsa/bin/ovsatool keygen -storekey -t ECDSA -n Intel -k custkeystore -r custkeystore.csr -e "/C=IN/CN=localhost" - + Below two files are created along with the keystore file: * ``custkeystore.csr`` - A Certificate Signing Request (CSR) * ``custkeystore.csr.crt``` - A self-signed certificate @@ -909,7 +909,7 @@ Step 2: Add a CA-Signed Certificate to a Key Store 4. Add the certificate to the key store: .. code-block:: sh - + /opt/ovsa/bin/ovsatool keygen -storecert -c custkeystore.csr.crt -k custkeystore Step 3: Request an access controlled Model from the Model Developer @@ -917,11 +917,11 @@ Step 3: Request an access controlled Model from the Model Developer This example uses scp to share data between the ovsa_runtime and ovsa_dev Guest VMs on the same Host Machine. -1. Communicate your need for a model to the Model Developer. The Developer will ask you to provide the certificate from your key store and other information. This example uses the length of time the model needs to be available. +1. Communicate your need for a model to the Model Developer. The Developer will ask you to provide the certificate from your key store and other information. This example uses the length of time the model needs to be available. 2. The model user's certificate needs to be provided to the Developer: .. code-block:: sh - + cd $OVSA_RUNTIME_ARTEFACTS scp custkeystore.csr.crt username@://OVSA/artefacts @@ -929,12 +929,12 @@ Step 4: Receive and load the access controlled model into the OpenVINO™ Model ------------------------------------------------------------------------------------ 1. Receive the model as files named: - + * face_detection_model.dat * face_detection_model.lic .. code-block:: sh - + cd $OVSA_RUNTIME_ARTEFACTS scp username@://OVSA/artefacts/face_detection_model.dat . scp username@://OVSA/artefacts/face_detection_model.lic . @@ -942,26 +942,26 @@ Step 4: Receive and load the access controlled model into the OpenVINO™ Model 2. Prepare the environment: .. code-block:: sh - + cd $OVSA_RUNTIME_ARTEFACTS/.. cp /opt/ovsa/example_runtime ovms -r cd ovms mkdir -vp model/fd/1 - - The ``$OVSA_RUNTIME_ARTEFACTS/../ovms`` directory contains scripts and a sample configuration JSON file to start the model server. + + The ``$OVSA_RUNTIME_ARTEFACTS/../ovms`` directory contains scripts and a sample configuration JSON file to start the model server. 3. Copy the artefacts from the Model Developer: .. code-block:: sh - + cd $OVSA_RUNTIME_ARTEFACTS/../ovms cp $OVSA_RUNTIME_ARTEFACTS/face_detection_model.dat model/fd/1/. cp $OVSA_RUNTIME_ARTEFACTS/face_detection_model.lic model/fd/1/. cp $OVSA_RUNTIME_ARTEFACTS/custkeystore model/fd/1/. - + 4. Rename and edit ``sample.json`` to include the names of the access controlled model artefacts you received from the Model Developer. The file looks like this: .. code-block:: sh - + { "custom_loader_config_list":[ { @@ -981,7 +981,7 @@ Step 4: Receive and load the access controlled model into the OpenVINO™ Model } ] } - + Step 5: Start the NGINX Model Server ------------------------------------ @@ -989,9 +989,9 @@ Step 5: Start the NGINX Model Server The NGINX Model Server publishes the access controlled model. .. code-block:: sh - + ./start_secure_ovsa_model_server.sh - + For information about the NGINX interface follow `here `__. Step 6: Prepare to run Inference @@ -1002,27 +1002,27 @@ Step 6: Prepare to run Inference 2. Install the Python dependencies for your set up. For example: .. code-block:: sh - + sudo apt install pip3 pip3 install cmake pip3 install scikit-build pip3 install opencv-python pip3 install futures==3.1.1 pip3 install tensorflow-serving-api==1.14.0 - + 3. Copy the ``face_detection.py`` from the example_client in ``/opt/ovsa/example_client`` .. code-block:: sh - + cd ~/OVSA/ovms cp /opt/ovsa/example_client/* . - + 4. Copy the sample images for inferencing. An image directory is created that includes a sample image for inferencing. .. code-block:: sh - + curl --create-dirs https://raw.githubusercontent.com/openvinotoolkit/model_server/master/example_client/images/people/people1.jpeg -o images/people1.jpeg - + Step 7: Run Inference --------------------- @@ -1030,7 +1030,7 @@ Step 7: Run Inference Run the ``face_detection.py`` script: .. code-block:: sh - + python3 face_detection.py --grpc_port 3335 --batch_size 1 --width 300 --height 300 --input_images_dir images --output_dir results --tls --server_cert /var/OVSA/Modelserver/server.pem --client_cert /var/OVSA/Modelserver/client.pem --client_key /var/OVSA/Modelserver/client.key --model_name controlled-access-model @@ -1039,8 +1039,8 @@ Summary You have completed these tasks: - Set up one or more computers (Host Machines) with one KVM per machine and one or more virtual machines (Guest VMs) on the Host Machines -- Installed the OpenVINO™ Security Add-on -- Used the OpenVINO™ Model Server to work with OpenVINO™ Security Add-on +- Installed the OpenVINO™ Security Add-on +- Used the OpenVINO™ Model Server to work with OpenVINO™ Security Add-on - As a Model Developer or Independent Software Vendor, you access controlled a model and prepared a license for it. - As a Model Developer or Independent Software Vendor, you prepared and ran a License Server and used the License Server to verify a User had a valid license to use a access controlled model. - As a User, you provided information to a Model Developer or Independent Software Vendor to get a access controlled model and the license for the model. diff --git a/docs/articles_en/documentation/openvino-ecosystem/openvino-training-extensions.rst b/docs/articles_en/documentation/openvino-ecosystem/openvino-training-extensions.rst index 4850fbc6fa2..4c5c2c847b2 100644 --- a/docs/articles_en/documentation/openvino-ecosystem/openvino-training-extensions.rst +++ b/docs/articles_en/documentation/openvino-ecosystem/openvino-training-extensions.rst @@ -5,15 +5,15 @@ OpenVINO™ Training Extensions .. meta:: - :description: OpenVINO™ Training Extensions include advanced algorithms used - to create, train and convert deep learning models with OpenVINO + :description: OpenVINO™ Training Extensions include advanced algorithms used + to create, train and convert deep learning models with OpenVINO Toolkit for optimized inference. OpenVINO™ Training Extensions provide a suite of advanced algorithms to train Deep Learning models and convert them using the `OpenVINO™ toolkit `__ for optimized -inference. It allows you to export and convert the models to the needed format. OpenVINO Training Extensions independently create and train the model. It is open-sourced and available on `GitHub `__. Read the OpenVINO Training Extensions `../../documentation `__ to learn more. +inference. It allows you to export and convert the models to the needed format. OpenVINO Training Extensions independently create and train the model. It is open-sourced and available on `GitHub `__. Read the OpenVINO Training Extensions `documentation `__ to learn more. Detailed Workflow ################# @@ -24,7 +24,7 @@ Detailed Workflow 2. OpenVINO Training Extensions train the model, using training interface, and evaluate the model quality on your dataset, using evaluation and inference interfaces. - .. note:: + .. note:: Prepare a separate dataset or split the dataset you have for more accurate quality evaluation. 3. Having successful evaluation results received, you have an opportunity to deploy your model or continue optimizing it, using NNCF. For more information about these frameworks, go to :doc:`Optimization Guide <../../openvino-workflow/model-optimization>`. @@ -34,9 +34,9 @@ If the results are unsatisfactory, add datasets and perform the same steps, star OpenVINO Training Extensions Components ####################################### -* `OpenVINO Training Extensions API `__ -* `OpenVINO Training Extensions CLI `__ -* `OpenVINO Training Extensions Algorithms `__ +* `OpenVINO Training Extensions API `__ +* `OpenVINO Training Extensions CLI `__ +* `OpenVINO Training Extensions Algorithms `__ Tutorials ######### diff --git a/docs/articles_en/openvino-workflow/deployment-locally.rst b/docs/articles_en/openvino-workflow/deployment-locally.rst index 569c49122dc..6bdbece9dd1 100644 --- a/docs/articles_en/openvino-workflow/deployment-locally.rst +++ b/docs/articles_en/openvino-workflow/deployment-locally.rst @@ -66,7 +66,7 @@ The granularity of OpenVINO packages may vary for different distribution types. - The main library ``openvino`` is used by users' C++ applications to link against with. For C language applications, ``openvino_c`` is additionally required for distribution. The library includes OpenVINO API 2.0. - The "optional" plugin libraries like ``openvino_intel_cpu_plugin`` (matching the ``openvino_.+_plugin`` pattern) are used to provide inference capabilities on specific devices or additional capabilities like :doc:`Hetero Execution ` and :doc:`Multi-Device Execution `. -- The "optional" plugin libraries like ``../documentation/openvino-ir-format_frontend`` (matching ``openvino_.+_frontend``) are used to provide capabilities to read models of different file formats such as OpenVINO IR, TensorFlow, ONNX, and PaddlePaddle. +- The "optional" plugin libraries like ``openvino_ir_frontend`` (matching ``openvino_.+_frontend``) are used to provide capabilities to read models of different file formats such as OpenVINO IR, TensorFlow, ONNX, and PaddlePaddle. Here the term "optional" means that if the application does not use the capability enabled by the plugin, the plugin library or a package with the plugin is not needed in the final distribution. diff --git a/docs/articles_en/openvino-workflow/deployment-locally/local-distribution-libraries.rst b/docs/articles_en/openvino-workflow/deployment-locally/local-distribution-libraries.rst index 9f3d8415129..ca471628f0e 100644 --- a/docs/articles_en/openvino-workflow/deployment-locally/local-distribution-libraries.rst +++ b/docs/articles_en/openvino-workflow/deployment-locally/local-distribution-libraries.rst @@ -114,17 +114,16 @@ Libraries for Execution Modes The ``HETERO``, ``MULTI``, ``BATCH``, and ``AUTO`` execution modes can also be used by the application explicitly or implicitly. Use the following recommendation scheme to decide whether to add the appropriate libraries to the distribution package: -- If :doc:`AUTO <../running-inference/inference-devices-and-modes/auto-device-selection>` is used explicitly in the application or `ov::Core::compile_model `__ is used without specifying a device, put ``openvino_auto_plugin`` to the distribution. +- If :doc:`AUTO <../running-inference/inference-devices-and-modes/auto-device-selection>` is used explicitly in the application or ov::Core::compile_model is used without specifying a device, put ``openvino_auto_plugin`` to the distribution. .. note:: Automatic Device Selection relies on :doc:`inference device plugins <../running-inference/inference-devices-and-modes>`. If you are not sure which inference devices are available on the target system, put all inference plugin libraries in the distribution. - If `ov::device::priorities `__ - is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only. + If ov::device::priorities is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only. - If :doc:`MULTI <../running-inference/inference-devices-and-modes/multi-device>` is used explicitly, put ``openvino_auto_plugin`` in the distribution. -- If :doc:`HETERO <../running-inference/inference-devices-and-modes/hetero-execution>` is either used explicitly or `ov::hint::performance_mode `__ is used with GPU, put ``openvino_hetero_plugin`` in the distribution. +- If :doc:`HETERO <../running-inference/inference-devices-and-modes/hetero-execution>` is either used explicitly or ov::hint::performance_mode is used with GPU, put ``openvino_hetero_plugin`` in the distribution. - If :doc:`BATCH <../running-inference/inference-devices-and-modes/automatic-batching>` is either used explicitly or ``ov::hint::performance_mode`` is used with GPU, put ``openvino_batch_plugin`` in the distribution. Frontend Libraries for Reading Models @@ -132,18 +131,18 @@ Frontend Libraries for Reading Models OpenVINO Runtime uses frontend libraries dynamically to read models in different formats: -- ``../../documentation/openvino-ir-format_frontend`` is used to read OpenVINO IR. +- ``openvino_ir_frontend`` is used to read OpenVINO IR. - ``openvino_tensorflow_frontend`` is used to read the TensorFlow file format. - ``openvino_tensorflow_lite_frontend`` is used to read the TensorFlow Lite file format. - ``openvino_onnx_frontend`` is used to read the ONNX file format. - ``openvino_paddle_frontend`` is used to read the Paddle file format. - ``openvino_pytorch_frontend`` is used to convert PyTorch model via ``openvino.convert_model`` API. -Depending on the model format types that are used in the application in `ov::Core::read_model `__, select the appropriate libraries. +Depending on the model format types that are used in the application in ov::Core::read_model, select the appropriate libraries. .. note:: - To optimize the size of the final distribution package, it is recommended to convert models to OpenVINO IR by using :doc:`model conversion API <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api>`. This way you do not have to keep TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, and other frontend libraries in the distribution package. + To optimize the size of the final distribution package, it is recommended to convert models to OpenVINO IR by using :doc:`model conversion API <../model-preparation>`. This way you do not have to keep TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, and other frontend libraries in the distribution package. Examples #################### @@ -152,16 +151,16 @@ Examples In this example, the application is written in C, performs inference on CPU, and reads models stored in the OpenVINO IR format. -The following libraries are used: ``openvino_c``, ``openvino``, ``openvino_intel_cpu_plugin``, and ``../../documentation/openvino-ir-format_frontend``. +The following libraries are used: ``openvino_c``, ``openvino``, ``openvino_intel_cpu_plugin``, and ``openvino_ir_frontend``. - The ``openvino_c`` library is a main dependency of the application. The app links against this library. - The ``openvino`` library is used as a private dependency for ``openvino_c`` and is also used in the deployment. - ``openvino_intel_cpu_plugin`` is used for inference. -- ``../../documentation/openvino-ir-format_frontend`` is used to read source models. +- ``openvino_ir_frontend`` is used to read source models. **MULTI execution on GPU and CPU in `tput` mode** -In this example, the application is written in C++, performs inference :doc:`simultaneously on GPU and CPU devices <../running-inference/inference-devices-and-modes/multi-device>` with the `ov::hint::PerformanceMode::THROUGHPUT `__ property set, and reads models stored in the ONNX format. +In this example, the application is written in C++, performs inference :doc:`simultaneously on GPU and CPU devices <../running-inference/inference-devices-and-modes/multi-device>` with the ov::hint::PerformanceMode::THROUGHPUT property set, and reads models stored in the ONNX format. The following libraries are used: ``openvino``, ``openvino_intel_gpu_plugin``, ``openvino_intel_cpu_plugin``, ``openvino_auto_plugin``, ``openvino_auto_batch_plugin``, and ``openvino_onnx_frontend``. diff --git a/docs/articles_en/openvino-workflow/deployment-locally/optimial-binary-size-conditional-compilation.rst b/docs/articles_en/openvino-workflow/deployment-locally/optimial-binary-size-conditional-compilation.rst index 0a2b28dc6f6..ed7604117a1 100644 --- a/docs/articles_en/openvino-workflow/deployment-locally/optimial-binary-size-conditional-compilation.rst +++ b/docs/articles_en/openvino-workflow/deployment-locally/optimial-binary-size-conditional-compilation.rst @@ -147,7 +147,7 @@ Put all CSV files together for ``stage 2`` to generate resulting OpenVINO binari Device-agnostic Conditional Compilation (POC) ############################################# -In some cases, adopting conditional compilation is necessary to support multiple different ``SKUs``(Stock Keeping Unit: is usually a string of numbers and alphabets used by the manufacturer to identify their product), but there may be limitations in collecting statistics information for every target hardware. To achieve this, conditional compilation must be capable of running a model on an accelerator with all previous SKUs. +In some cases, adopting conditional compilation is necessary to support multiple different ``SKUs`` (Stock Keeping Unit: is usually a string of numbers and alphabets used by the manufacturer to identify their product), but there may be limitations in collecting statistics information for every target hardware. To achieve this, conditional compilation must be capable of running a model on an accelerator with all previous SKUs. Conditional compilation requires the initial collection of statistical information to exclude unused code regions, such as ops and kernels. To do this, all included ops and kernels must be executed at least once. For multiple SKUs, it is necessary for all ops and kernels that will be used by any of the SKUs to be encountered at least once in the profiling data. If the profiling is done on a CPU platform, it is impossible without using an emulator. @@ -163,11 +163,7 @@ Considering that JIT kernels can be affected by L1/L2/L3 cache size and the numb - L2/L3 cache emulation - Hack the function of get cache size - - ``unsigned int dnnl::impl::cpu::platform::get_per_core_cache_size(int level)`` - - to make it return emulated cache size in analyzed stage, the simplest way is to leverage environment variable to pass the emulated cache size, for example: + Hack the function of get cache size ``unsigned int dnnl::impl::cpu::platform::get_per_core_cache_size(int level)`` to make it return emulated cache size in analyzed stage, the simplest way is to leverage environment variable to pass the emulated cache size, for example: .. code-block:: cpp diff --git a/docs/articles_en/openvino-workflow/generative-ai-models-guide.rst b/docs/articles_en/openvino-workflow/generative-ai-models-guide.rst index f9997264352..3914f31d5d2 100644 --- a/docs/articles_en/openvino-workflow/generative-ai-models-guide.rst +++ b/docs/articles_en/openvino-workflow/generative-ai-models-guide.rst @@ -170,15 +170,6 @@ For more details on compression options, refer to the :doc:`weight compression g OpenVINO also supports 4-bit models from Hugging Face `Transformers `__ library optimized with `GPTQ `__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it. -Another optimization that is applied by default when using ``OVModelForCausalLM`` class is transformation of the model to a stateful form. -This transformation further improves inference performance and decreases amount of allocated runtime memory in long running text generation scenarious. -It is achieved by hiding inputs and outputs of the model that represent past KV-cache tensors, and handling them inside the model in a more efficient way. -This feature is activated automatically for a wide range of supported text generation models, keeping not supported models in a regular, stateless form. - -Model usage are identical for stateful and stateless models as long as Optimum-Intel API is used because KV-cache handling is an internal detail of the text-generation API of Transformers library. -But a form of a model matterns in case when exported from Optimum-Intel OpenVINO model IR is used in an application implemented with native OpenVINO API, because stateful and stateless models have different number of inputs and outputs. -Please refer to a dedicated section of this document below for more information about using native OpenVINO API. - Below are some examples of using Optimum-Intel for model conversion and inference: * `Stable Diffusion v2.1 using Optimum-Intel OpenVINO `__ diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst index 962ff9121ee..ee7dc88f511 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst @@ -30,8 +30,8 @@ NNCF Quick Start Examples See the following Jupyter Notebooks for step-by-step examples showing how to add model compression to a PyTorch or Tensorflow training pipeline with NNCF: -- `Quantization Aware Training with NNCF and PyTorch `__. -- `Quantization Aware Training with NNCF and TensorFlow `__. +- `Quantization Aware Training with NNCF and PyTorch `__. +- `Quantization Aware Training with NNCF and TensorFlow `__. Installation #################### @@ -111,6 +111,6 @@ Additional Resources - :doc:`Quantizing Models Post-training ` - `NNCF GitHub repository `__ - `NNCF FAQ `__ -- `Quantization Aware Training with NNCF and PyTorch `__ -- `Quantization Aware Training with NNCF and TensorFlow `__. +- `Quantization Aware Training with NNCF and PyTorch `__. +- `Quantization Aware Training with NNCF and TensorFlow `__. diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/filter-pruning.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/filter-pruning.rst index 3275a0885aa..db26735aad1 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/filter-pruning.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/filter-pruning.rst @@ -7,23 +7,23 @@ Filter Pruning of Convolutional Models Introduction #################### -Filter pruning is an advanced optimization method that allows reducing the computational complexity of the model by removing -redundant or unimportant filters from the convolutional operations of the model. This removal is done in two steps: +Filter pruning is an advanced optimization method that allows reducing the computational complexity of the model by removing +redundant or unimportant filters from the convolutional operations of the model. This removal is done in two steps: 1. Unimportant filters are zeroed out by the NNCF optimization with fine-tuning. 2. Zero filters are removed from the model during the export to OpenVINO Intermediate Representation (IR). -Filter Pruning method from the NNCF can be used stand-alone but we usually recommend stacking it with 8-bit quantization for -two reasons. First, 8-bit quantization is the best method in terms of achieving the highest accuracy-performance trade-offs so -stacking it with filter pruning can give even better performance results. Second, applying quantization along with filter -pruning does not hurt accuracy a lot since filter pruning removes noisy filters from the model which narrows down values +Filter Pruning method from the NNCF can be used stand-alone but we usually recommend stacking it with 8-bit quantization for +two reasons. First, 8-bit quantization is the best method in terms of achieving the highest accuracy-performance trade-offs so +stacking it with filter pruning can give even better performance results. Second, applying quantization along with filter +pruning does not hurt accuracy a lot since filter pruning removes noisy filters from the model which narrows down values ranges of weights and activations and helps to reduce overall quantization error. .. note:: - Filter Pruning usually requires a long fine-tuning or retraining of the model which can be comparable to training the - model from scratch. Otherwise, a large accuracy degradation can be caused. Therefore, the training schedule should be + Filter Pruning usually requires a long fine-tuning or retraining of the model which can be comparable to training the + model from scratch. Otherwise, a large accuracy degradation can be caused. Therefore, the training schedule should be adjusted accordingly when applying this method. @@ -43,13 +43,13 @@ In this step, NNCF-related imports are added in the beginning of the training sc .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [imports] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -58,26 +58,26 @@ In this step, NNCF-related imports are added in the beginning of the training sc 2. Create NNCF configuration ++++++++++++++++++++++++++++ -Here, you should define NNCF configuration which consists of model-related parameters (`"input_info"` section) and parameters +Here, you should define NNCF configuration which consists of model-related parameters (`"input_info"` section) and parameters of optimization methods (`"compression"` section). .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [nncf_congig] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python :fragment: [nncf_congig] - -Here is a brief description of the required parameters of the Filter Pruning method. For a full description refer to the + +Here is a brief description of the required parameters of the Filter Pruning method. For a full description refer to the `GitHub `__ page. * ``pruning_init`` - initial pruning rate target. For example, value ``0.1`` means that at the begging of training, convolutions that can be pruned will have 10% of their filters set to zero. @@ -90,23 +90,23 @@ Here is a brief description of the required parameters of the Filter Pruning met 3. Apply optimization methods +++++++++++++++++++++++++++++ -In the next step, the original model is wrapped by the NNCF object using the ``create_compressed_model()`` API using the -configuration defined in the previous step. This method returns a so-called compression controller and the wrapped model -that can be used the same way as the original model. It is worth noting that optimization methods are applied at this step -so that the model undergoes a set of corresponding transformations and can contain additional operations required for the +In the next step, the original model is wrapped by the NNCF object using the ``create_compressed_model()`` API using the +configuration defined in the previous step. This method returns a so-called compression controller and the wrapped model +that can be used the same way as the original model. It is worth noting that optimization methods are applied at this step +so that the model undergoes a set of corresponding transformations and can contain additional operations required for the optimization. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [wrap_model] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -115,21 +115,21 @@ optimization. 4. Fine-tune the model ++++++++++++++++++++++ -This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the case -of Filter Pruning method we recommend using the training schedule and learning rate similar to what was used for the training +This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the case +of Filter Pruning method we recommend using the training schedule and learning rate similar to what was used for the training of the original model. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [tune_model] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python @@ -139,49 +139,49 @@ of the original model. 5. Multi-GPU distributed training +++++++++++++++++++++++++++++++++ -In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before the +In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before the fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [distributed] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python :fragment: [distributed] - + 6. Export quantized model +++++++++++++++++++++++++ -When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in +When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in the case of PyTorch and frozen graph - for TensorFlow 2. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [export] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python :fragment: [export] -These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model +These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model checkpoints during the training. Since NNCF wraps the original model with its own object it provides an API for these needs. @@ -194,18 +194,18 @@ To save model checkpoint use the following API: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [save_checkpoint] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python :fragment: [save_checkpoint] - + 8. (Optional) Restore from checkpoint +++++++++++++++++++++++++++++++++++++ @@ -216,31 +216,31 @@ To restore the model from checkpoint you should use the following API: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py :language: python :fragment: [load_checkpoint] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py :language: python - :fragment: [load_checkpoint] + :fragment: [load_checkpoint] -For more details on saving/loading checkpoints in the NNCF, see the following -`../../../documentation `__. +For more details on saving/loading checkpoints in the NNCF, see the following +`documentation `__. Deploying pruned model ###################### -The pruned model requres an extra step that should be done to get performance improvement. This step involves removal of the -zero filters from the model. This is done at the model conversion step using :doc:`model conversion API <../../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api>` tool when model is converted from the framework representation (ONNX, TensorFlow, etc.) to OpenVINO Intermediate Representation. +The pruned model requres an extra step that should be done to get performance improvement. This step involves removal of the +zero filters from the model. This is done at the model conversion step using :doc:`model conversion API <../../model-preparation>`>` tool when model is converted from the framework representation (ONNX, TensorFlow, etc.) to OpenVINO Intermediate Representation. * To remove zero filters from the pruned model add the following parameter to the model conversion command: ``transform=Pruning`` After that, the model can be deployed with OpenVINO in the same way as the baseline model. -For more details about model deployment with OpenVINO, see the corresponding :doc:`../../../documentation <../../running-inference>`. +For more details about model deployment with OpenVINO, see the corresponding :doc:`documentation <../../running-inference>`. Examples diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst index 85339940977..d39303396eb 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst @@ -7,20 +7,20 @@ Quantization-aware Training (QAT) Introduction #################### -Quantization-aware Training is a popular method that allows quantizing a model and applying fine-tuning to restore accuracy -degradation caused by quantization. In fact, this is the most accurate quantization method. This document describes how to -apply QAT from the Neural Network Compression Framework (NNCF) to get 8-bit quantized models. This assumes that you are +Quantization-aware Training is a popular method that allows quantizing a model and applying fine-tuning to restore accuracy +degradation caused by quantization. In fact, this is the most accurate quantization method. This document describes how to +apply QAT from the Neural Network Compression Framework (NNCF) to get 8-bit quantized models. This assumes that you are knowledgeable in Python programming and familiar with the training code for the model in the source DL framework. Using NNCF QAT #################### -Here, we provide the steps that are required to integrate QAT from NNCF into the training script written with +Here, we provide the steps that are required to integrate QAT from NNCF into the training script written with PyTorch or TensorFlow 2: .. note:: - Currently, NNCF for TensorFlow 2 supports optimization of the models created using Keras - `Sequential API `__ or + Currently, NNCF for TensorFlow 2 supports optimization of the models created using Keras + `Sequential API `__ or `Functional API `__. 1. Import NNCF API @@ -32,145 +32,145 @@ In this step, you add NNCF-related imports in the beginning of the training scri .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [imports] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python - :fragment: [imports] + :fragment: [imports] 2. Create NNCF configuration ++++++++++++++++++++++++++++ -Here, you should define NNCF configuration which consists of model-related parameters (``"input_info"`` section) and parameters -of optimization methods (``"compression"`` section). For faster convergence, it is also recommended to register a dataset object +Here, you should define NNCF configuration which consists of model-related parameters (``"input_info"`` section) and parameters +of optimization methods (``"compression"`` section). For faster convergence, it is also recommended to register a dataset object specific to the DL framework. It will be used at the model creation step to initialize quantization parameters. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [nncf_congig] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python - :fragment: [nncf_congig] - + :fragment: [nncf_congig] + 3. Apply optimization methods +++++++++++++++++++++++++++++ -In the next step, you need to wrap the original model object with the ``create_compressed_model()`` API using the configuration -defined in the previous step. This method returns a so-called compression controller and a wrapped model that can be used the -same way as the original model. It is worth noting that optimization methods are applied at this step so that the model -undergoes a set of corresponding transformations and can contain additional operations required for the optimization. In -the case of QAT, the compression controller object is used for model export and, optionally, in distributed training as it +In the next step, you need to wrap the original model object with the ``create_compressed_model()`` API using the configuration +defined in the previous step. This method returns a so-called compression controller and a wrapped model that can be used the +same way as the original model. It is worth noting that optimization methods are applied at this step so that the model +undergoes a set of corresponding transformations and can contain additional operations required for the optimization. In +the case of QAT, the compression controller object is used for model export and, optionally, in distributed training as it will be shown below. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [wrap_model] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python :fragment: [wrap_model] - + 4. Fine-tune the model ++++++++++++++++++++++ -This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the -case of QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle, +This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the +case of QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle, you can skip this step which means that the post-training optimization will be applied to the model. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [tune_model] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python :fragment: [tune_model] - + 5. Multi-GPU distributed training +++++++++++++++++++++++++++++++++ -In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before +In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before the fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [distributed] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python :fragment: [distributed] - + 6. Export quantized model +++++++++++++++++++++++++ -When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in +When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in the case of PyTorch and frozen graph - for TensorFlow 2. .. tab-set:: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [export] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python :fragment: [export] - + .. note:: - The precision of weights gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation. + The precision of weights gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation. You can expect the model footprint reduction only for that format. -These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model +These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model checkpoints during the training. Since NNCF wraps the original model with its own object it provides an API for these needs. 7. (Optional) Save checkpoint @@ -182,13 +182,13 @@ To save model checkpoint use the following API: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [save_checkpoint] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python @@ -204,26 +204,26 @@ To restore the model from checkpoint you should use the following API: .. tab-item:: PyTorch :sync: pytorch - + .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py :language: python :fragment: [load_checkpoint] - + .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 + :sync: tensorflow-2 .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py :language: python :fragment: [load_checkpoint] - -For more details on saving/loading checkpoints in the NNCF, see the following `../../../documentation `__. + +For more details on saving/loading checkpoints in the NNCF, see the following `documentation `__. Deploying quantized model ######################### -The quantized model can be deployed with OpenVINO in the same way as the baseline model. No extra steps or options are -required in this case. For more details, see the corresponding :doc:`../../../documentation <../../running-inference>`. +The quantized model can be deployed with OpenVINO in the same way as the baseline model. No extra steps or options are +required in this case. For more details, see the corresponding :doc:`documentation <../../running-inference>`. Examples #################### diff --git a/docs/articles_en/openvino-workflow/model-preparation.rst b/docs/articles_en/openvino-workflow/model-preparation.rst index 1485fb82341..b408bb1e09c 100644 --- a/docs/articles_en/openvino-workflow/model-preparation.rst +++ b/docs/articles_en/openvino-workflow/model-preparation.rst @@ -96,8 +96,7 @@ Convert a Model with Python: ``convert_model`` ############################################## Model Conversion API in Python uses the ``openvino.convert_model`` function, to turn a model -into the `openvino.Model `__ -object and load it to memory. Now it can be: saved to a drive with ``openvino.save_model`` +into the openvino.Model object and load it to memory. Now it can be: saved to a drive with ``openvino.save_model`` or further :doc:`optimized with NNCF ` before saving. diff --git a/docs/articles_en/openvino-workflow/model-preparation/conversion-parameters.rst b/docs/articles_en/openvino-workflow/model-preparation/conversion-parameters.rst index bf184f279c1..d18bfb32c46 100644 --- a/docs/articles_en/openvino-workflow/model-preparation/conversion-parameters.rst +++ b/docs/articles_en/openvino-workflow/model-preparation/conversion-parameters.rst @@ -7,7 +7,7 @@ Conversion Parameters .. meta:: :description: Model Conversion API provides several parameters to adjust model conversion. -This document describes all available parameters for ``openvino.convert_model``, ``ovc``, and ``openvino.save_model`` without focusing on a particular framework model format. Use this information for your reference as a common description of the conversion API capabilities in general. Part of the options can be not relevant to some specific frameworks. Use :doc:`Supported Model Formats <../model-preparation>` page for more dedicated framework-dependent ../../learn-openvino/interactive-tutorials-python. +This document describes all available parameters for ``openvino.convert_model``, ``ovc``, and ``openvino.save_model`` without focusing on a particular framework model format. Use this information for your reference as a common description of the conversion API capabilities in general. Part of the options can be not relevant to some specific frameworks. Use :doc:`Supported Model Formats <../model-preparation>` page for more dedicated framework-dependent tutorials. In most cases when it is required to convert a model the following simple syntax can be used: diff --git a/docs/articles_en/openvino-workflow/model-preparation/convert-model-paddle.rst b/docs/articles_en/openvino-workflow/model-preparation/convert-model-paddle.rst index d9c0f706786..de0bcf07110 100644 --- a/docs/articles_en/openvino-workflow/model-preparation/convert-model-paddle.rst +++ b/docs/articles_en/openvino-workflow/model-preparation/convert-model-paddle.rst @@ -144,5 +144,5 @@ For the list of supported standard layers, refer to the :doc:`Supported Operatio Additional Resources #################### -Check out more examples of model conversion in :doc:`interactive Python ../../learn-openvino/interactive-tutorials-python <../../learn-openvino/interactive-tutorials-python>`. +Check out more examples of model conversion in :doc:`interactive Python tutorials <../../learn-openvino/interactive-tutorials-python>`. diff --git a/docs/articles_en/openvino-workflow/model-preparation/convert-model-pytorch.rst b/docs/articles_en/openvino-workflow/model-preparation/convert-model-pytorch.rst index c9e5eb08863..ab59219929d 100644 --- a/docs/articles_en/openvino-workflow/model-preparation/convert-model-pytorch.rst +++ b/docs/articles_en/openvino-workflow/model-preparation/convert-model-pytorch.rst @@ -84,7 +84,7 @@ In practice, the code to evaluate or test the PyTorch model is usually provided category_name = weights.meta["categories"][class_id] print(f"{category_name}: {100 * score:.1f}% (with OpenVINO)") -Check out more examples in :doc:`interactive Python ../../learn-openvino/interactive-tutorials-python <../../learn-openvino/interactive-tutorials-python>`. +Check out more examples in :doc:`interactive Python tutorials <../../learn-openvino/interactive-tutorials-python>`. .. note:: @@ -121,10 +121,10 @@ If your model has a ``dict`` input, such as, ``{"x": a, "y": b, "z": c}``, it wi An important consequence of flattening is that only ``tuple`` and ``dict`` with a fixed number of elements and key values are supported. The structure of such inputs should be fully described in the ``example_input`` parameter of ``convert_model``. The flattening on outputs should be reproduced with the given ``example_input`` and cannot be changed once the conversion is done. -Check out more examples of model conversion with non-tensor data types in the following ../../learn-openvino/interactive-tutorials-python: +Check out more examples of model conversion with non-tensor data types in the following tutorials: -* `Video Subtitle Generation using Whisper and OpenVINO™ `__ -* `Visual Question Answering and Image Captioning using BLIP and OpenVINO `__ +* `Video Subtitle Generation using Whisper and OpenVINO™ `__ +* `Visual Question Answering and Image Captioning using BLIP and OpenVINO `__ Exporting a PyTorch Model to ONNX Format diff --git a/docs/articles_en/openvino-workflow/model-preparation/convert-model-to-ir.rst b/docs/articles_en/openvino-workflow/model-preparation/convert-model-to-ir.rst index 08e49ddfb10..65fc98341a7 100644 --- a/docs/articles_en/openvino-workflow/model-preparation/convert-model-to-ir.rst +++ b/docs/articles_en/openvino-workflow/model-preparation/convert-model-to-ir.rst @@ -56,8 +56,8 @@ Here are code examples of how to use these methods with different model formats: compiled_model = core.compile_model(ov_model, "AUTO") For more details on conversion, refer to the - :doc:`guide <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-pytorch>` - and an example `tutorial `__ + :doc:`guide ` + and an example `tutorial `__ on this topic. .. tab-item:: TensorFlow @@ -98,8 +98,8 @@ Here are code examples of how to use these methods with different model formats: compiled_model = core.compile_model(ov_model, "AUTO") For more details on conversion, refer to the - :doc:`guide <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-tensorflow>` - and an example `tutorial `__ + :doc:`guide ` + and an example `tutorial `__ on this topic. * The ``read_model()`` and ``compile_model()`` methods: @@ -172,7 +172,7 @@ Here are code examples of how to use these methods with different model formats: ovc .pb For details on the conversion, refer to the - :doc:`article <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-tensorflow>`. + :doc:`article <../model-preparation>`. .. tab-item:: TensorFlow Lite :sync: tflite @@ -199,8 +199,8 @@ Here are code examples of how to use these methods with different model formats: compiled_model = core.compile_model(ov_model, "AUTO") For more details on conversion, refer to the - :doc:`guide <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-tensorflow>` - and an example `tutorial `__ + :doc:`guide ` + and an example `tutorial `__ on this topic. @@ -290,7 +290,7 @@ Here are code examples of how to use these methods with different model formats: ovc .tflite For details on the conversion, refer to the - :doc:`article <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-tensorflow-lite>`. + :doc:`article `. .. tab-item:: ONNX :sync: onnx @@ -317,8 +317,8 @@ Here are code examples of how to use these methods with different model formats: compiled_model = core.compile_model(ov_model, "AUTO") For more details on conversion, refer to the - :doc:`guide <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-onnx>` - and an example `tutorial `__ + :doc:`guide ` + and an example `tutorial `__ on this topic. @@ -385,7 +385,7 @@ Here are code examples of how to use these methods with different model formats: ov_compiled_model_t* compiled_model = NULL; ov_core_compile_model_from_file(core, ".onnx", "AUTO", 0, &compiled_model); - For details on the conversion, refer to the :doc:`article <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-onnx>` + For details on the conversion, refer to the :doc:`article ` .. tab-item:: CLI :sync: cli @@ -405,7 +405,7 @@ Here are code examples of how to use these methods with different model formats: ovc .onnx For details on the conversion, refer to the - :doc:`article <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-onnx>` + :doc:`article ` .. tab-item:: PaddlePaddle :sync: pdpd @@ -438,8 +438,8 @@ Here are code examples of how to use these methods with different model formats: compiled_model = core.compile_model(ov_model, "AUTO") For more details on conversion, refer to the - :doc:`guide <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-paddle>` - and an example `tutorial `__ + :doc:`guide ` + and an example `tutorial `__ on this topic. * The ``read_model()`` method: @@ -527,7 +527,7 @@ Here are code examples of how to use these methods with different model formats: ovc .pdmodel For details on the conversion, refer to the - :doc:`article <../../documentation/legacy-features/transition-legacy-conversion-api/legacy-conversion-api/[legacy]-supported-model-formats/[legacy]-convert-paddle>`. + :doc:`article `. * :doc:`How to convert PyTorch `