natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Docs benchmarks update for 2024.0 (#23231) 

### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
Signed-off-by: Andrew Park <andrew.park@intel.com>
Signed-off-by: hyunback <hyunback.kim@intel.com>
Signed-off-by: Min, Byungil <byungil.min@intel.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: HyungKi Jeong <68764729+testdrive-profiling-master@users.noreply.github.com>
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Georgy Krivoruchko <georgy.krivoruchko@intel.com>
Co-authored-by: Tomasz Jankowski <tomasz1.jankowski@intel.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
Co-authored-by: River Li <river.li@intel.com>
Co-authored-by: Pavel Durandin <pavel.durandin@intel.com>
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
Co-authored-by: Fang Xu <fang.xu@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Luwei Zhou <luwei.zhou@intel.com>
Co-authored-by: cecilia peng <cecilia.peng@intel.com>
Co-authored-by: Maksim Kutakov <maxim.kutakov@gmail.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
Co-authored-by: Mateusz Tabaka <mateusz.tabaka@intel.com>
Co-authored-by: Daniil Lyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Anastasia Kuporosova <anastasia.kuporosova@intel.com>
Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>
Co-authored-by: Jorge Ragde <ragdejorge@gmail.com>
Co-authored-by: Vitaliy Urusovskij <vitaliy.urusovskij@intel.com>
Co-authored-by: Oleg Pipikin <oleg.pipikin@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
Co-authored-by: Egor Duplenskii <egor.duplensky@gmail.com>
Co-authored-by: Raasz, Pawel <pawel.raasz@intel.com>
Co-authored-by: Xiping Yan <xiping.yan@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Xuejun Zhai <xuejun.zhai@intel.com>
Co-authored-by: Andrew Kwangwoong Park <andrew.park@intel.com>
Co-authored-by: Luo Cheng <cheng.luo@intel.com>
Co-authored-by: Zhang Yi <yi3.zhang@intel.com>
Co-authored-by: Sun Xiaoxia <xiaoxia.sun@intel.com>
Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Co-authored-by: Sergey Shlyapnikov <sergey.shlyapnikov@intel.com>
Co-authored-by: Taylor Yeonbok Lee <taylor.lee@intel.com>
Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: Evgeny Kotov <evgeny.kotov@intel.com>
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
Co-authored-by: Xiuchuan Zhai <xiuchuan.zhai@intel.com>
Co-authored-by: Jade Cho <jade.cho@intel.com>
Co-authored-by: hyunback kim <hyunback.kim@intel.com>
Co-authored-by: Wilson Seok <wilson.seok@intel.com>
Co-authored-by: Maksim Kutakov <maksim.kutakov@intel.com>
Co-authored-by: Vladislav Golubev <vladislav.golubev@intel.com>
Co-authored-by: Min, Byungil <byungil.min@intel.com>
Co-authored-by: Tingqian Li <tingqian.li@intel.com>
Co-authored-by: Evgenya Nugmanova <evgeniia.nugmanova@intel.com>
Co-authored-by: Anastasiia Pnevskaia <anastasiia.pnevskaia@intel.com>
Co-authored-by: Edward Shogulin <edward.shogulin@intel.com>
Co-authored-by: Irina Efode <irina.efode@intel.com>
Co-authored-by: Ekaterina Aidova <ekaterina.aidova@intel.com>
Co-authored-by: Eddy Kim <eddy.kim@intel.com>
Co-authored-by: Kelvin Choi <kelvin.choi@intel.com>
Co-authored-by: Steve Yoo <steve.yoo@intel.com>
Co-authored-by: Roman Lyamin <Roman.Lyamin@intel.com>
Co-authored-by: David Nam <david.nam@intel.com>
Co-authored-by: Alexandra Sidorova <alexandra.sidorova@intel.com>
Co-authored-by: Yury Gaydaychuk <yury.gaydaychuk@intel.com>
Co-authored-by: Denis Orlov <denis.orlov@intel.com>
Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Mingyu Kim <mingyu.kim@intel.com>
Co-authored-by: Sebastian Golebiewski <sebastianx.golebiewski@intel.com>
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nikolay Shchegolev <nikolay.shchegolev@intel.com>
Co-authored-by: Piotr Kowalczyk <piotr.kowalczyk@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
Co-authored-by: Surya Siddharth Pemmaraju <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: ynimmaga <yamini.nimmagadda@intel.com>
Co-authored-by: Sofya Balandina <sofya.balandina@intel.com>
Co-authored-by: Nesterov Alexander <alexander.nesterov@intel.com>
Co-authored-by: Evgeny Talanin <evgeny.talanin@intel.com>
Co-authored-by: Piotr Krzemiński <piotr.krzeminski@intel.com>
Co-authored-by: Gorokhov Dmitriy <dmitry.gorokhov@intel.com>
Co-authored-by: Pavel Esir <pavel.esir@intel.com>
Co-authored-by: Łukasz Mielicki <lukasz.mielicki@intel.com>
Co-authored-by: Vladimir Paramuzov <vladimir.paramuzov@intel.com>
Co-authored-by: Jacek Pawlak <jacek.pawlak@intel.com>
Co-authored-by: tadamczx <156996781+tadamczx@users.noreply.github.com>
Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
Co-authored-by: Alexander Suvorov <alexander.suvorov@intel.com>
This commit is contained in:
Andrzej Kopytko 2024-03-05 13:56:30 +01:00 committed by GitHub
parent 5253f01bea
commit edbdd39806
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 528 additions and 713 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/articles_en/about-openvino/performance-benchmarks.rst
View File

@ -119,21 +119,21 @@ For a listing of all platforms and configurations used for testing, refer to the
.. grid-item::
.. button-link:: ../_static/benchmarks_files/OV-2023.3-platform_list.pdf
.. button-link:: ../_static/benchmarks_files/OV-2024.0-platform_list.pdf
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Hardware Platforms [PDF]
.. button-link:: ../_static/benchmarks_files/OV-2023.3-system-info-detailed.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. button-link:: ../_static/benchmarks_files/OV-2023.3-Performance-Data.xlsx
.. button-link:: ../_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
:color: primary
:outline:
:expand:
@ -192,7 +192,7 @@ or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devc
Disclaimers
####################################
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2023.3, as of February 13, 2024.
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2024.0, as of March 06, 2024.
* OpenVINO Model Server performance results are based on release 2023.3, as of February 13, 2024.

155
docs/articles_en/about-openvino/performance-benchmarks/model-accuracy-int8-fp32.rst
View File

@ -5,19 +5,17 @@ Model Accuracy
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
the table for more information.
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
the table for more information.
* A - Intel® Core™ i9-9000K (AVX2), INT8 and FP32
* B - Intel® Xeon® 6338, (VNNI), INT8 and FP32
* C - Intel(R) Xeon 8490H (VNNI, AMX), INT8, BF16, FP32
* D - Intel® Flex-170, INT8 and FP16
.. list-table:: Model Accuracy for INT8
:header-rows: 1
@ -31,10 +29,10 @@ the table for more information.
* - bert-base-cased
- SST-2_bert_cased_padded
- spearman@cosine
- 3.17%
- 3.28%
- 2.68%
- 3.00%
- 2.73%
- 2.91%
- 2.72%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
@ -46,16 +44,16 @@ the table for more information.
- COCO2017_detection_91cl
- coco_precision
- -0.84%
- -0.59%
- -0.64%
- -0.62%
- -0.63%
* - mask_rcnn_resnet50_atrous_coco
- COCO2017_detection_91cl_bkgr
- coco_orig_precision
- 0.03%
- 0.08%
- 0.11%
- 0.07%
- -0.04%
- 0.02%
- 0.04%
- 0.04%
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
@ -66,16 +64,16 @@ the table for more information.
* - resnet-50
- ImageNet2012
- accuracy @ top1
- -0.20%
- -0.19%
- -0.09%
- -0.12%
- -0.13%
- -0.15%
- -0.19%
* - ssd-resnet34-1200
- COCO2017_detection_80cl_bkgr
- map
- -0.03%
- -0.06%
- -0.02%
- -0.01%
- -0.02%
- 0.04%
* - ssd-mobilenet-v1-coco
- COCO2017_detection_80cl_bkgr
@ -94,45 +92,45 @@ the table for more information.
* - yolo_v3_tiny
- COCO2017_detection_80cl
- map
- %
- -0.23%
- -0.24%
- -0.66%
- -0.30%
- -0.43%
- -0.43%
- -0.87%
* - yolo_v8n
- COCO2017_detection_80cl
- map
- -0.02%
- -0.03%
- -0.06%
- -0.06%
- -0.01%
- -0.04%
- 0.04%
- -0.08%
* - chatGLM2-6b
- lambada openai
- ppl
-
- 17.38
- 17.41
- 17.17
-
- 0.75
- 0.75
-
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
- 3.24
- 3.24
- 3.25
-
- 3.38
- 3.27
-
* - Stable-Diffusion-V2-1
- LIAON-5B
- CLIP
-
-
-
-
-
-
-
* - Mistral-7b
- proprietary Mistral.ai
- ppl
-
- 3.29
- 3.47
-
- 3.49
- 3.19
-
.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
:header-rows: 1
@ -151,16 +149,16 @@ the table for more information.
- 0.00%
- 0.00%
- 0.00%
- -0.09%
- 0.00%
- -0.03%
- 0.01%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
- 0.04%
- 0.04%
- 0.04%
- 0.06%
- 0.04%
- 0.05%
- 0.05%
* - efficientdet-d0
- COCO2017_detection_91cl
- coco_precision
@ -173,9 +171,9 @@ the table for more information.
- COCO2017_detection_91cl_bkgr
- coco_orig_precision
- -0.01%
- -0.01%
- -0.02%
- %
- -0.18%
- 0.09%
- 0.02%
* - mobilenet-v2
- ImageNet2012
@ -183,16 +181,16 @@ the table for more information.
- 0.00%
- 0.00%
- 0.00%
- -0.04%
- -0.18%
- 0.02%
* - resnet-50
- ImageNet2012
- accuracy @ top1
- 0.02%
- 0.02%
- 0.00%
- 0.01%
- 0.01%
- 0.00%
- 0.00%
- -0.01%
- -0.01%
* - ssd-resnet34-1200
- COCO2017_detection_80cl_bkgr
- map
@ -207,8 +205,8 @@ the table for more information.
- 0.01%
- 0.01%
- 0.01%
- 0.05%
- -0.03%
- 0.04%
- -0.04%
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
@ -220,52 +218,53 @@ the table for more information.
* - yolo_v3_tiny
- COCO2017_detection_80cl
- map
- %
- 0.00%
- 0.00%
- 0.00%
- -0.02%
- 0.25%
- -0.01%
* - yolo_v8n
- COCO2017_detection_80cl
- map
- 0.00%
- 0.00%
- 0.00%
- 0.05%
- -0.03%
- 0.04%
- -0.02%
* - chatGLM2-6b
- lambada openai
- ppl
-
- 17.48
- 17.56
-
- 17.49
-
- 0.75
- 0.8
-
-
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
-
- 3.26
- 3.26
-
-
-
* - Stable-Diffusion-V2-1
- LIAON-5B
- CLIP
-
-
-
-
- 22.48
-
-
-
-
-
* - Mistral-7b
- proprietary Mistral.ai
- ppl
-
- 3.19
-
- 3.18
-
-
- 3.19
-
-
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
accuracy for the model.
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
accuracy for the model.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2023.3-platform_list.pdf
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2023.3-system-info-detailed.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2023.3-Performance-Data.xlsx → docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-Performance-Data.xlsx
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-platform_list.pdf Normal file
View File

Binary file not shown.

BIN
docs/sphinx_setup/_static/benchmarks_files/OV-2024.0-system-info-detailed.xlsx Normal file
View File

Binary file not shown.

1078
docs/sphinx_setup/_static/benchmarks_files/OV-benchmark-data.csv
View File

File diff suppressed because it is too large Load Diff