diff --git a/docs/Documentation/deployment_guide_introduction.md b/docs/Documentation/deployment_guide_introduction.md index 69ee5cfdabe..96d3e874a2e 100644 --- a/docs/Documentation/deployment_guide_introduction.md +++ b/docs/Documentation/deployment_guide_introduction.md @@ -6,29 +6,25 @@ :maxdepth: 1 :hidden: - Run Inference - Inference Optimization + Deploy via OpenVINO Runtime + Deploy via Model Serving -.. toctree:: - :maxdepth: 1 - :hidden: - - Deploy Locally - Deploy Using Model Server +@endsphinxdirective -Once you have a model that meets both OpenVINO™ and your requirements, you can choose among several ways of deploying it with your application. The two default options are: +Once you have a model that meets both OpenVINO™ and your requirements, you can choose how to deploy it with your application. +@sphinxdirective .. panels:: - `Deploying locally `_ + `Deploy Locally `_ ^^^^^^^^^^^^^^ - Local deployment simply uses OpenVINO Runtime installed on the device. It utilizes resources available to the system. + Local deployment uses OpenVINO Runtime installed on the device. It utilizes resources available to the system and provides the quickest way of launching inference. --- - `Deploying by Model Serving `_ + `Deploy by Model Serving `_ ^^^^^^^^^^^^^^ Deployment via OpenVINO Model Server allows the device to connect to the server set up remotely. This way inference uses external resources instead of the ones provided by the device itself. @@ -36,6 +32,5 @@ Once you have a model that meets both OpenVINO™ and your requirements, you can @endsphinxdirective -> **NOTE**: Note that [running inference in OpenVINO Runtime](../OV_Runtime_UG/openvino_intro.md) is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration. Inference may be additionally optimized, as described in the [Inference Optimization section](../optimization_guide/dldt_deployment_optimization_guide.md). Apart from the default deployment options, you may also [deploy your application for the TensorFlow framework with OpenVINO Integration](./openvino_ecosystem_ovtf.md). \ No newline at end of file diff --git a/docs/Documentation/openvino_ecosystem.md b/docs/Documentation/openvino_ecosystem.md index f274432bdf2..0d3f78ae2d4 100644 --- a/docs/Documentation/openvino_ecosystem.md +++ b/docs/Documentation/openvino_ecosystem.md @@ -6,9 +6,9 @@ :maxdepth: 1 :hidden: - ovsa_get_started ovtf_integration ote_documentation + ovsa_get_started openvino_inference_engine_tools_compile_tool_README openvino_docs_tuning_utilities workbench_docs_Workbench_DG_Introduction diff --git a/docs/Documentation/openvino_workflow.md b/docs/Documentation/openvino_workflow.md new file mode 100644 index 00000000000..96068cdf407 --- /dev/null +++ b/docs/Documentation/openvino_workflow.md @@ -0,0 +1,53 @@ +# OPENVINO Workflow {#openvino_workflow} + + +@sphinxdirective + +.. toctree:: + :maxdepth: 1 + :hidden: + + Model Preparation + Model Optimization and Compression + Deployment + +@endsphinxdirective + + + +THIS IS A PAGE ABOUT THE WORKFLOW + +@sphinxdirective + +.. raw:: html + +
+ + +
+
+
+ + link to model processing introduction + +
+
+ +
+
+ + link to an optimization guide + +
+
+ +
+
+ + link to deployment introduction + +
+
+ + +@endsphinxdirective \ No newline at end of file diff --git a/docs/OV_Runtime_UG/deployment/deployment_intro.md b/docs/OV_Runtime_UG/deployment/deployment_intro.md index 9d179082462..023fd0a5c30 100644 --- a/docs/OV_Runtime_UG/deployment/deployment_intro.md +++ b/docs/OV_Runtime_UG/deployment/deployment_intro.md @@ -1,4 +1,4 @@ -# Deploying Your Applications with OpenVINO™ {#openvino_deployment_guide} +# Deploy via OpenVINO Runtime {#openvino_deployment_guide} @sphinxdirective @@ -6,14 +6,15 @@ :maxdepth: 1 :hidden: - openvino_docs_install_guides_deployment_manager_tool - openvino_docs_deploy_local_distribution + Run Inference + Optimize Inference + Deploy Application with Deployment Manager + Local Distribution Libraries @endsphinxdirective -Once [OpenVINO™ application development](../integrate_with_your_application.md) has been finished, application developers usually need to deploy their applications to end users. There are several ways to achieve that. This section will explain how you can deploy locally, using OpenVINO Runtime. +> **NOTE**: Note that [running inference in OpenVINO Runtime](../openvino_intro.md) is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration and [develop your application properly](../integrate_with_your_application.md) -> **NOTE**: Note that [running inference in OpenVINO Runtime](../openvino_intro.md) is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration. ## Local Deployment Options diff --git a/docs/OV_Runtime_UG/openvino_intro.md b/docs/OV_Runtime_UG/openvino_intro.md index b33cffd5f25..50aa0e3d504 100644 --- a/docs/OV_Runtime_UG/openvino_intro.md +++ b/docs/OV_Runtime_UG/openvino_intro.md @@ -12,9 +12,7 @@ openvino_docs_Runtime_Inference_Modes_Overview openvino_docs_OV_UG_Working_with_devices openvino_docs_OV_UG_ShapeInference - openvino_docs_OV_UG_Preprocessing_Overview openvino_docs_OV_UG_DynamicShapes - openvino_docs_OV_UG_Performance_Hints openvino_docs_OV_UG_network_state_intro @endsphinxdirective diff --git a/docs/OV_Runtime_UG/performance_hints.md b/docs/OV_Runtime_UG/performance_hints.md index 1561947cc02..5b825b7d43a 100644 --- a/docs/OV_Runtime_UG/performance_hints.md +++ b/docs/OV_Runtime_UG/performance_hints.md @@ -9,7 +9,7 @@ Previously, a certain level of automatic configuration was the result of the *de The hints, in contrast, respect the actual model, so the parameters for optimal throughput are calculated for each model individually (based on its compute versus memory bandwidth requirements and capabilities of the device). ## Performance Hints: Latency and Throughput -As discussed in the [Optimization Guide](../optimization_guide/dldt_optimization_guide.md) there are a few different metrics associated with inference speed. +As discussed in the [Optimization Guide](../optimization_guide/dldt_deployment_optimization_guide.md) there are a few different metrics associated with inference speed. Throughput and latency are some of the most widely used metrics that measure the overall performance of an application. Therefore, in order to ease the configuration of the device, OpenVINO offers two dedicated hints, namely `ov::hint::PerformanceMode::THROUGHPUT` and `ov::hint::PerformanceMode::LATENCY`. diff --git a/docs/OV_Runtime_UG/supported_plugins/CPU.md b/docs/OV_Runtime_UG/supported_plugins/CPU.md index aa178fd5088..c0626f025a8 100644 --- a/docs/OV_Runtime_UG/supported_plugins/CPU.md +++ b/docs/OV_Runtime_UG/supported_plugins/CPU.md @@ -298,5 +298,5 @@ To enable denormals optimization in the application, the `denormals_optimization ## Additional Resources * [Supported Devices](Supported_Devices.md) -* [Optimization guide](@ref openvino_docs_optimization_guide_dldt_optimization_guide) +* [Optimization guide](@ref openvino_docs_deployment_optimization_guide_dldt_optimization_guide) * [СPU plugin developers documentation](https://github.com/openvinotoolkit/openvino/wiki/CPUPluginDevelopersDocs) diff --git a/docs/OV_Runtime_UG/supported_plugins/GPU.md b/docs/OV_Runtime_UG/supported_plugins/GPU.md index 25c18694747..7aff81594b4 100644 --- a/docs/OV_Runtime_UG/supported_plugins/GPU.md +++ b/docs/OV_Runtime_UG/supported_plugins/GPU.md @@ -309,5 +309,5 @@ Since OpenVINO relies on the OpenCL kernels for the GPU implementation, many gen ## Additional Resources * [Supported Devices](Supported_Devices.md) -* [Optimization guide](@ref openvino_docs_optimization_guide_dldt_optimization_guide) +* [Optimization guide](@ref openvino_docs_deployment_optimization_guide_dldt_optimization_guide) * [GPU plugin developers documentation](https://github.com/openvinotoolkit/openvino/wiki/GPUPluginDevelopersDocs) diff --git a/docs/benchmarks/performance_benchmarks_faq.md b/docs/benchmarks/performance_benchmarks_faq.md index 2a5ff070be1..e73280ea3c4 100644 --- a/docs/benchmarks/performance_benchmarks_faq.md +++ b/docs/benchmarks/performance_benchmarks_faq.md @@ -23,11 +23,11 @@ All of the performance benchmarks are generated using the open-source tool within the Intel® Distribution of OpenVINO™ toolkit called `benchmark_app`. This tool is available - `for C++ apps `_ + `for C++ apps `_ as well as - `for Python apps`_. + `for Python apps `_. - For a simple instruction on testing performance, see the `Getting Performance Numbers Guide` + For a simple instruction on testing performance, see the `Getting Performance Numbers Guide `_. .. dropdown:: What image sizes are used for the classification network models? @@ -107,16 +107,16 @@ .. dropdown:: Where can I purchase the specific hardware used in the benchmarking? Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the - [Intel® AI: In Production Partners & Solutions Catalog](https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html). + `Intel® AI: In Production Partners & Solutions Catalog `_. For more details, see the [Supported Devices](../OV_Runtime_UG/supported_plugins/Supported_Devices.md) documentation. Before purchasing any hardware, you can test and run - models remotely, using [Intel® DevCloud for the Edge](http://devcloud.intel.com/edge/). + models remotely, using `Intel® DevCloud for the Edge `_. .. dropdown:: How can I optimize my models for better performance or accuracy? Set of guidelines and recommendations to optimize models are available in the - [optimization guide](../optimization_guide/dldt_optimization_guide.md). - Join the conversation in the [Community Forum](https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit) + [optimization guide](../optimization_guide/dldt_deployment_optimization_guide.md). + Join the conversation in the `Community Forum ` for further support. .. dropdown:: Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support? diff --git a/docs/documentation.md b/docs/documentation.md index e89db7cbd67..ab8da3afa65 100644 --- a/docs/documentation.md +++ b/docs/documentation.md @@ -7,21 +7,11 @@ :hidden: API Reference - Model Preparation - Model Optimization and Compression - Deployment Tool Ecosystem OpenVINO Extensibility Media Processing and CV Libraries OpenVINO™ Security -.. toctree:: - :maxdepth: 1 - :hidden: - - openvino_docs_optimization_guide_dldt_optimization_guide - - @endsphinxdirective diff --git a/docs/home.rst b/docs/home.rst index 09e9aceab0f..b3709fe668a 100644 --- a/docs/home.rst +++ b/docs/home.rst @@ -40,7 +40,7 @@ A typical workflow with OpenVINO is shown below.
- + link to an optimization guide
@@ -122,6 +122,7 @@ Get Started GET STARTED LEARN OPENVINO + OPENVINO WORKFLOW DOCUMENTATION MODEL ZOO RESOURCES diff --git a/docs/optimization_guide/dldt_deployment_optimization_guide.md b/docs/optimization_guide/dldt_deployment_optimization_guide.md index 5acfebcf9dc..5626900fcb3 100644 --- a/docs/optimization_guide/dldt_deployment_optimization_guide.md +++ b/docs/optimization_guide/dldt_deployment_optimization_guide.md @@ -1,4 +1,4 @@ -# Runtime Inference Optimization {#openvino_docs_deployment_optimization_guide_dldt_optimization_guide} +# Optimize Inference {#openvino_docs_deployment_optimization_guide_dldt_optimization_guide} @sphinxdirective @@ -7,9 +7,11 @@ :hidden: openvino_docs_deployment_optimization_guide_common + openvino_docs_OV_UG_Performance_Hints openvino_docs_deployment_optimization_guide_latency openvino_docs_deployment_optimization_guide_tput openvino_docs_deployment_optimization_guide_tput_advanced + openvino_docs_OV_UG_Preprocessing_Overview openvino_docs_deployment_optimization_guide_internals @endsphinxdirective diff --git a/docs/optimization_guide/dldt_deployment_optimization_latency.md b/docs/optimization_guide/dldt_deployment_optimization_latency.md index 11781230d31..7cc93b2e4fc 100644 --- a/docs/optimization_guide/dldt_deployment_optimization_latency.md +++ b/docs/optimization_guide/dldt_deployment_optimization_latency.md @@ -1,4 +1,4 @@ -## Optimizing for the Latency {#openvino_docs_deployment_optimization_guide_latency} +## Optimizing for Latency {#openvino_docs_deployment_optimization_guide_latency} @sphinxdirective diff --git a/docs/optimization_guide/dldt_optimization_guide.md b/docs/optimization_guide/dldt_optimization_guide.md deleted file mode 100644 index cdc2c006606..00000000000 --- a/docs/optimization_guide/dldt_optimization_guide.md +++ /dev/null @@ -1,36 +0,0 @@ -# Introduction to Performance Optimization {#openvino_docs_optimization_guide_dldt_optimization_guide} -Even though inference performance should be defined as a combination of many factors, including accuracy and efficiency, it is most often described as the speed of execution. As the rate with which the model processes live data, it is based on two fundamentally interconnected metrics: latency and throughput. - - - -![](../img/LATENCY_VS_THROUGHPUT.svg) - -**Latency** measures inference time (in ms) required to process a single input. When it comes to executing multiple inputs simultaneously (for example, via batching), the overall throughput (inferences per second, or frames per second, FPS, in the specific case of visual processing) is usually more of a concern. -**Throughput** is calculated by dividing the number of inputs that were processed by the processing time. - -## End-to-End Application Performance -It is important to separate the "pure" inference time of a neural network and the end-to-end application performance. For example, data transfers between the host and a device may unintentionally affect the performance when a host input tensor is processed on the accelerator such as dGPU. - -Similarly, the input-preprocessing contributes significantly to the inference time. As described in the [getting performance numbers](../MO_DG/prepare_model/Getting_performance_numbers.md) section, when evaluating *inference* performance, one option is to measure all such items separately. -For the **end-to-end scenario**, though, consider image pre-processing with OpenVINO and the asynchronous execution as a way to lessen the communication costs (like data transfers). For more details, see the [general optimizations guide](./dldt_deployment_optimization_common.md). - -Another specific case is **first-inference latency** (for example, when a fast application start-up is required), where the resulting performance may be well dominated by the model loading time. [Model caching](../OV_Runtime_UG/Model_caching_overview.md) may be considered as a way to improve model loading/compilation time. - -Finally, **memory footprint** restriction is another possible concern when designing an application. While this is a motivation for the use of the *model* optimization techniques, keep in mind that the throughput-oriented execution is usually much more memory consuming. For more details, see the [Runtime Inference Optimizations guide](../optimization_guide/dldt_deployment_optimization_guide.md). - - -> **NOTE**: To get performance numbers for OpenVINO, along with the tips on how to measure and compare it with a native framework, see the [Getting performance numbers article](../MO_DG/prepare_model/Getting_performance_numbers.md). - -## Improving Performance: Model vs Runtime Optimizations - -> **NOTE**: First, make sure that your model can be successfully inferred with OpenVINO Runtime. - -There are two primary optimization approaches to improving inference performance with OpenVINO: model- and runtime-level optimizations. They are **fully compatible** and can be done independently. - -- **Model optimizations** include model modifications, such as quantization, pruning, optimization of preprocessing, etc. For more details, refer to this [document](./model_optimization_guide.md). - - The model optimizations directly improve the inference time, even without runtime parameters tuning (described below). - -- **Runtime (Deployment) optimizations** includes tuning of model *execution* parameters. Fore more details, see the [Runtime Inference Optimizations guide](../optimization_guide/dldt_deployment_optimization_guide.md). - -## Performance benchmarks -A wide range of public models for estimating performance and comparing the numbers (measured on various supported devices) are available in the [Performance benchmarks section](../benchmarks/performance_benchmarks.md). \ No newline at end of file diff --git a/docs/tutorials.md b/docs/tutorials.md index eb3bd18da56..7db634222c8 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -1,4 +1,4 @@ -# Tutorials {#tutorials} +# Interactive Tutorials (Python) {#tutorials} @sphinxdirective diff --git a/tools/legacy/benchmark_app/README.md b/tools/legacy/benchmark_app/README.md index 4651447ce7b..5e4bebc5899 100644 --- a/tools/legacy/benchmark_app/README.md +++ b/tools/legacy/benchmark_app/README.md @@ -53,11 +53,8 @@ Note that the benchmark_app usually produces optimal performance for any device ./benchmark_app -m -i -d CPU ``` -But it is still may be sub-optimal for some cases, especially for very small networks. More details can read in [Performance Optimization Guide](../../../docs/optimization_guide/dldt_optimization_guide.md). - -As explained in the [Performance Optimization Guide](../../../docs/optimization_guide/dldt_optimization_guide.md) section, for all devices, including new [MULTI device](../../../docs/OV_Runtime_UG/supported_plugins/MULTI.md) it is preferable to use the FP16 IR for the model. -Also if latency of the CPU inference on the multi-socket machines is of concern, please refer to the same -[Performance Optimization Guide](../../../docs/optimization_guide/dldt_optimization_guide.md). +It still may be sub-optimal for some cases, especially for very small networks. For all devices, including the [MULTI device](../../../docs/OV_Runtime_UG/supported_plugins/MULTI.md) it is preferable to use the FP16 IR for the model. If latency of the CPU inference on the multi-socket machines is of concern. +These, as well as other topics are explained in the [Performance Optimization Guide](../../../docs/optimization_guide/dldt_deployment_optimization_guide.md). Running the application with the `-h` option yields the following usage message: ```