From c2b15154b4c91ea84f52577cfedcf009c5594a27 Mon Sep 17 00:00:00 2001 From: Maciej Smyk Date: Tue, 11 Jun 2024 10:54:42 +0200 Subject: [PATCH] [DOCS] torch.compile documentation updates for 2024.2 (#24908) Port from https://github.com/openvinotoolkit/openvino/pull/24803 ### Details: - Updated 'How to Use' and 'Options' sections - Added quantization and torchserve sections --------- Co-authored-by: Sebastian Golebiewski --- .../openvino-workflow/torch-compile.rst | 83 +++++++++++++++++-- 1 file changed, 77 insertions(+), 6 deletions(-) diff --git a/docs/articles_en/openvino-workflow/torch-compile.rst b/docs/articles_en/openvino-workflow/torch-compile.rst index 57682f2e143..f8edc79fe51 100644 --- a/docs/articles_en/openvino-workflow/torch-compile.rst +++ b/docs/articles_en/openvino-workflow/torch-compile.rst @@ -22,20 +22,33 @@ By default, Torch code runs in eager-mode, but with the use of ``torch.compile`` How to Use #################### -To use ``torch.compile``, you need to add an import statement and define the ``openvino`` backend. +To use ``torch.compile``, you need to define the ``openvino`` backend in your PyTorch application. This way Torch FX subgraphs will be directly converted to OpenVINO representation without any additional PyTorch-based tracing/scripting. +This approach works only for the **package distributed via pip**, as it is now configured with +`torch_dynamo_backends entrypoint `__. +.. code-block:: python -.. code-block:: sh - - import openvino.torch ... model = torch.compile(model, backend='openvino') + ... + +For OpenVINO installed via channels other than pip, such as conda, and versions older than +2024.1, an additional import statement is needed: + +.. code-block:: python + + import openvino.torch + + ... + model = torch.compile(model, backend='openvino') + ... Execution diagram: .. image:: ../assets/images/torch_compile_backend_openvino.svg + :alt: torch.compile execution diagram :width: 992px :height: 720px :scale: 60% @@ -51,6 +64,9 @@ enable model caching, set the cache directory etc. You can use a dictionary of t By default, the OpenVINO backend for ``torch.compile`` runs PyTorch applications on CPU. If you set this variable to ``GPU.0``, for example, the application will use the integrated graphics processor instead. +* ``aot_autograd`` - enables a graph capture needed for dynamic shapes or to finetune a + model. For models with dynamic shapes, it is recommended to set this option to ``True``. + By default, aot_autograd is set to ``False``. * ``model_caching`` - enables saving the optimized model files to a hard drive, after the first application run. This makes them available for the following application executions, reducing the first-inference latency. By default, this @@ -58,6 +74,15 @@ enable model caching, set the cache directory etc. You can use a dictionary of t * ``cache_dir`` - enables defining a custom directory for the model files (if ``model_caching`` is set to ``True``). By default, the OpenVINO IR is saved in the cache sub-directory, created in the application's root directory. +* ``decompositions`` - enables defining additional operator decompositions. By + default, this is an empty list. For example, to add a decomposition for + an operator ``my_op``, add ``'decompositions': [torch.ops.aten.my_op.default]`` + to the options. +* ``disabled_ops`` - enables specifying operators that can be disabled from + openvino execution and make it fall back to native PyTorch runtime. For + example, to disable an operator ``my_op`` from OpenVINO execution, add + ``'disabled_ops': [torch.ops.aten.my_op.default]`` to the options. By + default, this is an empty list. * ``config`` - enables passing any OpenVINO configuration option as a dictionary to this variable. For details on the various options, refer to the :ref:`OpenVINO Advanced Features `. @@ -79,8 +104,10 @@ You can also set OpenVINO specific configuration options by adding them as a dic Windows support +++++++++++++++++++++ -Currently, PyTorch does not support ``torch.compile`` feature on Windows officially. However, it can be accessed by running -the below instructions: +PyTorch supports ``torch.compile`` officially on Windows from version 2.3.0 onwards. + +For PyTorch versions below 2.3.0, the ``torch.compile`` feature is not supported on Windows +officially. However, it can be accessed by running the following instructions: 1. Install the PyTorch nightly wheel file - `2.1.0.dev20230713 `__ , 2. Update the file at ``/Lib/site-packages/torch/_dynamo/eval_frames.py`` @@ -104,6 +131,50 @@ the below instructions: if sys.version_info >= (3, 11): `raise RuntimeError("Python 3.11+ not yet supported for torch.compile") +Support for PyTorch 2 export quantization (Preview) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +PyTorch 2 export quantization is supported by OpenVINO backend in ``torch.compile``. To be able +to access this feature, follow the steps provided in +`PyTorch 2 Export Post Training Quantization with X86 Backend through Inductor `__ +and update the provided sample as explained below. + +1. If you are using PyTorch version 2.3.0 or later, disable constant folding in quantization to + be able to benefit from the optimization in the OpenVINO backend. This can be done by passing + ``fold_quantize=False`` parameter into the ``convert_pt2e`` function. To do so, change this + line: + + .. code-block:: python + + converted_model = convert_pt2e(prepared_model) + + to the following: + + .. code-block:: python + + converted_model = convert_pt2e(prepared_model, fold_quantize=False) + +2. Set ``torch.compile`` backend as OpenVINO and execute the model. + + Update this line below: + + .. code-block:: python + + optimized_model = torch.compile(converted_model) + + As below: + + .. code-block:: python + + optimized_model = torch.compile(converted_model, backend="openvino") + +TorchServe Integration ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +TorchServe is a performant, flexible, and easy to use tool for serving PyTorch models in production. For more information on the details of TorchServe, +you can refer to `TorchServe github repository. `__. With OpenVINO ``torch.compile`` integration into TorchServe you can serve +PyTorch models in production and accelerate them with OpenVINO on various Intel hardware. Detailed instructions on how to use OpenVINO with TorchServe are +available in `TorchServe examples. `__ Support for Automatic1111 Stable Diffusion WebUI +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++