diff --git a/docs/articles_en/openvino_workflow/model_preparation.rst b/docs/articles_en/openvino_workflow/model_preparation.rst index f9bc4680666..a70d79c5233 100644 --- a/docs/articles_en/openvino_workflow/model_preparation.rst +++ b/docs/articles_en/openvino_workflow/model_preparation.rst @@ -15,7 +15,6 @@ Model Preparation Convert to OpenVINO Model Conversion Parameters Setting Input Shapes - PyVision preprocessing You can obtain a model in one of supported formats, **PyTorch, TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle**, diff --git a/docs/articles_en/openvino_workflow/model_preparation/pytorch_vision.rst b/docs/articles_en/openvino_workflow/model_preparation/pytorch_vision.rst deleted file mode 100644 index 34df9e465e0..00000000000 --- a/docs/articles_en/openvino_workflow/model_preparation/pytorch_vision.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. {#pytorch_vision} - -PyVision -======================= - - -.. meta:: - :description: Learn about supported model formats and the methods used to convert, read, and compile them in OpenVINO™. - -Images input to AI models often need to be preprocessed in order to have proper dimensions or data type. -Instead of doing it with another library in an additional pipeline step, you can use torchvision.transforms OpenVINO feature. -It automatically translates a torchvision preprocessing pipeline to OpenVINO operators and then embeds them into your OpenVINO model, reducing overall program complexity and allowing additional performance optimizations to take place. \ No newline at end of file diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide.rst index 2628e8be39c..5c389e3d789 100644 --- a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide.rst +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide.rst @@ -13,10 +13,11 @@ Optimize Inference openvino_docs_OV_UG_Precision_Control openvino_docs_deployment_optimization_guide_latency openvino_docs_deployment_optimization_guide_tput - openvino_docs_deployment_optimization_guide_tput_advanced + Advanced Throughput Options openvino_docs_OV_UG_Preprocessing_Overview openvino_docs_deployment_optimization_guide_internals - openvino_docs_memory_optimization_guide + Optimizing memory usage + .. meta:: :description: Improving inference performance involves model and runtime diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput.rst index 3f1595189ac..b26d0c9e613 100644 --- a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput.rst +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput.rst @@ -10,8 +10,15 @@ Optimizing for Throughput simultaneously which improves the device utilization. -As described in the section on the :doc:`latency-specific considerations `, one of the possible use cases is *delivering every single request at the minimal delay*. -Throughput, on the other hand, is about inference scenarios in which potentially **large number of inference requests are served simultaneously to improve the device utilization**. +.. toctree:: + :maxdepth: 1 + :hidden: + + Advanced Throughput Options + + +As described in the section on the :doc:`latency-specific optimizations `, one of the possible use cases is delivering every single request with minimal delay. +Throughput, on the other hand, is about inference scenarios in which potentially **large numbers of inference requests are served simultaneously to improve resource use**. The associated increase in latency is not linearly dependent on the number of requests executed in parallel. A trade-off between overall throughput and serial performance of individual requests can be achieved with the right performance configuration of OpenVINO. diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput_advanced.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput_advanced.rst index eff5a21ae1c..b792cfca42b 100644 --- a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput_advanced.rst +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/dldt_deployment_optimization_tput_advanced.rst @@ -1,6 +1,6 @@ .. {#openvino_docs_deployment_optimization_guide_tput_advanced} -Using Advanced Throughput Options: Streams and Batching +Advanced Throughput Options: Streams and Batching ======================================================= @@ -8,7 +8,7 @@ Using Advanced Throughput Options: Streams and Batching :description: With OpenVINO streams a device may handle processing multiple inference requests and the batching helps to saturate the device and leads to higher throughput. - + OpenVINO Streams #################### diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/precision_control.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/precision_control.rst index 91ae9b011f5..63de6309809 100644 --- a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/precision_control.rst +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/precision_control.rst @@ -9,9 +9,8 @@ The choice of data types is essential to the inference runtime, which can have a 1. Model storage precision (IR precision), 2. Model inference precision. -Previously, these 2 precisions were interrelated, and model storage precision could affect the inference precision in some devices (e.g. GPU did ``f16`` inference only for ``f16`` IRs). - -With the ``2023.0`` release this behavior has been changed and the inference precision no longer depends on the precision of IR. Now users have several knobs to find the balance between model performance and accuracy. +Inference precision no longer depends on the precision of IR, which means that users +have several options to find the balance between model performance and accuracy. Essentially, the IR precision becomes a way of compressing the model by reducing the precision of the weights, and it does not affect how the devices execute the model. This change clears up a lot of confusion where, for example, you couldn't execute a high-performance model on the GPU by default, and the behavior between devices was different. diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview.rst index bac4584fb4c..48020d3928d 100644 --- a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview.rst +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview.rst @@ -11,6 +11,7 @@ Optimize Preprocessing openvino_docs_OV_UG_Preprocessing_Details openvino_docs_OV_UG_Layout_Overview openvino_docs_OV_UG_Preprocess_Usecase_save + Torchvision preprocessing converter .. meta:: :description: The preprocessing entails additional operations to transform diff --git a/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview/torchvision_preprocessing_converter.rst b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview/torchvision_preprocessing_converter.rst new file mode 100644 index 00000000000..264edda073b --- /dev/null +++ b/docs/articles_en/openvino_workflow/running_inference_with_openvino/dldt_deployment_optimization_guide/preprocessing_overview/torchvision_preprocessing_converter.rst @@ -0,0 +1,71 @@ +.. {#torchvision_preprocessing_converter} + +Torchvision preprocessing converter +======================================= + + +.. meta:: + :description: See how OpenVINO™ enables torchvision preprocessing + to optimize model inference. + + +The Torchvision-to-OpenVINO converter enables automatic translation of operators from the torchvision +preprocessing pipeline to the OpenVINO format and embed them in your model. It is often used to adjust +images serving as input for AI models to have proper dimensions or data types. + +As the converter is fully based on the **openvino.preprocess** module, you can implement the **torchvision.transforms** +feature easily and without the use of external libraries, reducing the overall application complexity +and enabling additional performance optimizations. + + +.. note:: + + Not all torchvision transforms are supported yet. The following operations are available: + + .. code-block:: + + transforms.Compose + transforms.Normalize + transforms.ConvertImageDtype + transforms.Grayscale + transforms.Pad + transforms.ToTensor + transforms.CenterCrop + transforms.Resize + + +Example +################### + +.. code-block:: py + + preprocess_pipeline = torchvision.transforms.Compose( + [ + torchvision.transforms.Resize(256, interpolation=transforms.InterpolationMode.NEAREST), + torchvision.transforms.CenterCrop((216, 218)), + torchvision.transforms.Pad((2, 3, 4, 5), fill=3), + torchvision.transforms.ToTensor(), + torchvision.transforms.ConvertImageDtype(torch.float32), + torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), + ] + ) + + torch_model = SimpleConvnet(input_channels=3) + + torch.onnx.export(torch_model, torch.randn(1, 3, 224, 224), "test_convnet.onnx", verbose=False, input_names=["input"], output_names=["output"]) + core = Core() + ov_model = core.read_model(model="test_convnet.onnx") + + test_input = np.random.randint(255, size=(260, 260, 3), dtype=np.uint16) + ov_model = PreprocessConverter.from_torchvision( + model=ov_model, transform=preprocess_pipeline, input_example=Image.fromarray(test_input.astype("uint8"), "RGB") + ) + ov_model = core.compile_model(ov_model, "CPU") + ov_input = np.expand_dims(test_input, axis=0) + output = ov_model.output(0) + ov_result = ov_model(ov_input)[output] + + + + + diff --git a/docs/dev/get_started.md b/docs/dev/get_started.md index 93b6000dbee..b5d09af43dd 100644 --- a/docs/dev/get_started.md +++ b/docs/dev/get_started.md @@ -18,4 +18,4 @@ Explore other resources to learn more about OpenVINO: * [OpenVINO Developer Documentation](./index.md) * [OpenVINO Samples](../../samples) * [OpenVINO Building Documentation](./building_documentation.md) - * [CMake Options for Custom Compilation](./cmake_options_for_custom_comiplation.md) + * [CMake Options for Custom Compilation](./cmake_options_for_custom_compilation.md)