natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] benchmark pages 24.1 (#24144) 

Co-authored-by: Andrzej Kopytko <andrzejx.kopytko@intel.com>
This commit is contained in:
Karol Blaszczak 2024-04-23 11:41:46 +02:00 committed by GitHub
parent f5e1214487
commit 599c63a925
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
10 changed files with 632 additions and 569 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

135
docs/articles_en/about-openvino/performance-benchmarks.rst
View File

@ -17,10 +17,12 @@ Performance Benchmarks
performance-benchmarks/getting-performance-numbers
This page presents benchmark results for `Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
and :doc:`OpenVINO Model Server <../ovms_what_is_openvino_model_server>`, for a representative selection of public neural networks and Intel® devices.
The results may help you decide which hardware to use in your applications or plan AI workload for the hardware you have already implemented in your solutions.
Click the buttons below to see the chosen benchmark data.
This page presents benchmark results for
`Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
and :doc:`OpenVINO Model Server <../ovms_what_is_openvino_model_server>`, for a representative
selection of public neural networks and Intel® devices. The results may help you decide which
hardware to use in your applications or plan AI workload for the hardware you have already
implemented in your solutions. Click the buttons below to see the chosen benchmark data.
.. grid:: 1 1 2 2
:gutter: 4
@ -66,36 +68,41 @@ Please visit the tabs below for more information on key performance indicators a
throughput per dollar of system cost is the best measure of value. The value KPI is
calculated as “Throughput measured as inferences per second / price of inference engine”.
This means for a 2 socket system 2x the price of a CPU is used. Prices are as per
date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
date of benchmarking and sources can be found as links in the Hardware Platforms (PDF)
description below.
.. tab-item:: Efficiency
:sync: efficiency
System power is a key consideration from the edge to the data center. When selecting
deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider.
Intel designs provide excellent power efficiency for running deep learning workloads.
The efficiency KPI is calculated as “Throughput measured as inferences per second / TDP of
inference engine”. This means for a 2 socket system 2x the power dissipation (TDP) of a CPU is used.
TDP-values are as per date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
deep learning solutions, power efficiency (throughput/watt) is a critical factor to
consider. Intel designs provide excellent power efficiency for running deep learning
workloads. The efficiency KPI is calculated as “Throughput measured as inferences per
second / TDP of inference engine”. This means for a 2 socket system 2x the power
dissipation (TDP) of a CPU is used. TDP-values are as per date of benchmarking and sources
can be found as links in the Hardware Platforms (PDF) description below.
.. tab-item:: Latency
:sync: latency
This measures the synchronous execution of inference requests and is reported in milliseconds.
Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before
the next is started. This performance metric is relevant in usage scenarios where a single image
input needs to be acted upon as soon as possible. An example would be the healthcare sector where
medical personnel only request analysis of a single ultra sound scanning image or in real-time or
near real-time applications for example an industrial robot's response to actions in its environment
or obstacle avoidance for autonomous vehicles.
This measures the synchronous execution of inference requests and is reported in
milliseconds. Each inference request (for example: preprocess, infer, postprocess) is
allowed to complete before the next is started. This performance metric is relevant in
usage scenarios where a single image input needs to be acted upon as soon as possible. An
example would be the healthcare sector where medical personnel only request analysis of a
single ultra sound scanning image or in real-time or near real-time applications for
example an industrial robot's response to actions in its environment or obstacle avoidance
for autonomous vehicles.
.. tab-item:: Workload Parameters
:sync: workloadparameters
The workload parameters affect the performance results of the different models we use for benchmarking.
Image processing models have different image size definitions and the Natural Language Processing models
have different max token list lengths. All these can be found in detail in the :doc:`FAQ section <performance-benchmarks/performance-benchmarks-faq>`.
All models are executed using a batch size of 1. Below are the parameters for the GenAI models we display.
The workload parameters affect the performance results of the different models we use for
benchmarking. Image processing models have different image size definitions and the
Natural Language Processing models have different max token list lengths. All these can
be found in detail in the :doc:`FAQ section <performance-benchmarks/performance-benchmarks-faq>`.
All models are executed using a batch size of 1. Below are the parameters for the GenAI
models we display.
* Input tokens: 1024,
* Output tokens: 128,
@ -119,21 +126,21 @@ For a listing of all platforms and configurations used for testing, refer to the
.. grid-item::
.. button-link:: ../_static/benchmarks_files/OV-2024.0-platform_list.pdf
.. button-link:: ../_static/benchmarks_files/OV-2024.1-platform_list.pdf
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Hardware Platforms [PDF]
.. button-link:: ../_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.1-system-info-detailed.xlsx
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. button-link:: ../_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.1-Performance-Data.xlsx
:color: primary
:outline:
:expand:
@ -141,15 +148,16 @@ For a listing of all platforms and configurations used for testing, refer to the
:material-regular:`download;1.5em` Click for Performance Data [XLSX]
The OpenVINO benchmark setup includes a single system with OpenVINO™, as well as the benchmark application installed.
It measures the time spent on actual inference (excluding any pre or post processing) and then reports on the inferences
per second (or Frames Per Second).
The OpenVINO benchmark setup includes a single system with OpenVINO™, as well as the benchmark
application installed. It measures the time spent on actual inference (excluding any pre or post
processing) and then reports on the inferences per second (or Frames Per Second).
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime libraries and exposes a set of
models via a convenient inference API over gRPC or HTTP/REST. Its benchmark results are measured with the configuration of
multiple-clients-single-server, using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The connection is dedicated
only to measuring performance.
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime
libraries and exposes a set of models via a convenient inference API over gRPC or HTTP/REST.
Its benchmark results are measured with the configuration of multiple-clients-single-server,
using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The
connection is dedicated only to measuring performance.
.. dropdown:: See more details about OVMS benchmark setup
@ -158,22 +166,28 @@ only to measuring performance.
.. image:: ../_static/images/performance_benchmarks_ovms_02.png
:alt: OVMS Benchmark Setup Diagram
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on)
requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application
in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into
the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it
listens to (and answers) requests from clients. OpenVINO™ Model Server is run on the same
system as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models
served by OpenVINO™ Model Server are located in a local file system mounted into the docker
container. The OpenVINO™ Model Server instance communicates with other components via ports
over a dedicated docker network.
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3
programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™
Model Server before it will send a new next request. The role played by the clients is also verification of responses.
* **Clients** are run in separated physical machine referred to as client platform. Clients
are implemented in Python3 programming language based on TensorFlow* API and they work as
parallel processes. Each client waits for a response from OpenVINO™ Model Server before it
will send a new next request. The role played by the clients is also verification of
responses.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is
counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by
Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes
impact of physical network on reported metrics.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for
this purpose. Its main role is counting of requests forwarded from clients to OpenVINO™
Model Server, estimating its latency, and sharing this information by Prometheus service.
The reason of locating the load balancer on the client site is to simulate real life
scenario that includes impact of physical network on reported metrics.
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process,
downloading metrics from the load balancer, and presenting the final report of the execution.
* **Execution Controller** is launched on the client platform. It is responsible for
synchronization of the whole measurement process, downloading metrics from the load
balancer, and presenting the final report of the execution.
@ -181,12 +195,16 @@ only to measuring performance.
<h2>Test performance yourself</h2>
You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers <performance-benchmarks/getting-performance-numbers>`.
You can also test performance for your system yourself, following the guide on
:doc:`getting performance numbers <performance-benchmarks/getting-performance-numbers>`.
Performance of a particular application can also be evaluated virtually using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit.
To learn more about it, visit `the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
Performance of a particular application can also be evaluated virtually using
`Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions
of the Intel® Distribution of the OpenVINO™ Toolkit. To learn more about it, visit
`the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or
`create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
.. raw:: html
@ -194,17 +212,22 @@ or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devc
<h2>Disclaimers</h2>
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2024.0, as of March 06, 2024.
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release
2024.1, as of April 17, 2024.
* OpenVINO Model Server performance results are based on release 2024.0, as of March 15, 2024.
* OpenVINO Model Server performance results are based on release
2024.0, as of March 15, 2024.
The results may not reflect all publicly available updates. Intel technologies' features and benefits depend on system configuration
and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
The results may not reflect all publicly available updates. Intel technologies' features and
benefits depend on system configuration and may require enabled hardware, software, or service
activation. Learn more at intel.com, or from the OEM or retailer.
See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at `www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
Performance varies by use, configuration and other factors. Learn more at
`www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
Your costs and results may vary.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree
for non-Intel products.

42
docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst
View File

@ -29,16 +29,16 @@ the table for more information.
* - bert-base-cased
- SST-2_bert_cased_padded
- spearman@cosine
- 3.28%
- 2.93%
- 2.68%
- 2.91%
- 2.76%
- 2.72%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
- 0.07%
- 0.19%
- -0.03%
- 0.13%
- 0.03%
- 0.11%
* - efficientdet-d0
- COCO2017_detection_91cl
@ -57,7 +57,7 @@ the table for more information.
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
- %
-
- -0.97%
- -0.97%
- -0.95%
@ -85,9 +85,9 @@ the table for more information.
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
- -6.32%
- 6.40%
- -6.28%
- 6.41%
- 6.46%
- 6.40%
* - yolo_v3_tiny
- COCO2017_detection_80cl
@ -131,6 +131,13 @@ the table for more information.
- 3.49
- 3.19
-
* - Falcon-7b-instruct
- Bai Ze (65%), GPT4All (25%), GPTeacher (5%), RefinedWeb-english (5%)
- ppl
-
-
-
-
.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
:header-rows: 1
@ -157,7 +164,7 @@ the table for more information.
- 0.04%
- 0.04%
- 0.04%
- 0.05%
- 0.06%
- 0.05%
* - efficientdet-d0
- COCO2017_detection_91cl
@ -172,9 +179,9 @@ the table for more information.
- coco_orig_precision
- -0.01%
- -0.02%
- %
- -0.01%
- 0.09%
- 0.02%
- 0.00%
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
@ -206,7 +213,7 @@ the table for more information.
- 0.01%
- 0.01%
- 0.04%
- -0.04%
- -0.02%
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
@ -251,8 +258,8 @@ the table for more information.
- LIAON-5B
- CLIP
-
-
-
- 31.3
- 22.4
-
-
* - Mistral-7b
@ -263,7 +270,14 @@ the table for more information.
- 3.19
-
-
* - Falcon-7b-instruct
- Bai Ze (65%), GPT4All (25%), GPTeacher (5%), RefinedWeb-english (5%)
- ppl
-
-
-
-
-
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured

10
docs/articles_en/about-openvino/performance-benchmarks/performance-benchmarks-faq.rst
View File

@ -32,7 +32,6 @@ Performance Information F.A.Q.
All of the performance benchmarks are generated using the
open-source tool within the Intel® Distribution of OpenVINO™ toolkit
called :doc:`benchmark_app <../../learn-openvino/openvino-samples/benchmark-tool>`.
This tool is available for Python and C++ apps.
For a simple instruction on testing performance, see the :doc:`Getting Performance Numbers Guide <getting-performance-numbers>`.
@ -53,6 +52,10 @@ Performance Information F.A.Q.
- THUDM
- Transformer
- 32K
* - `Falcon-7b-instruct <https://huggingface.co/tiiuae/falcon-7b-instruct>`__
- Hugginface
- Causal Decoder-only
- 2048
* - `Llama-2-7b-chat <https://ai.meta.com/llama/>`__
- Meta AI
- Auto regressive language
@ -115,8 +118,9 @@ Performance Information F.A.Q.
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
Before purchasing any hardware, you can test and run models remotely, using
`Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
For more details, see the :doc:`Supported Devices <../compatibility-and-support/supported-devices>`.
documentation. Before purchasing any hardware, you can test and run
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
.. dropdown:: How can I optimize my models for better performance or accuracy?

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-platform_list.pdf
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-Performance-Data.xlsx Normal file
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-platform_list.pdf Normal file
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.1-system-info-detailed.xlsx Normal file
View File

Binary file not shown.

1014
docs/sphinx_setup/_static/benchmarks_files/OV-benchmark-data.csv
View File

File diff suppressed because it is too large Load Diff